Hello,
We talked a bit about the building, the electrical network, the towers and the servers for the BETA but "what's new" at a network level in BHS?
It's advancing. And every week it gets better and better!
We currently have three pairs of fibres between our DC in BHS and Montreal. We're managing them and we have 2x80Gbps lit. We decided to create our main Montreal POP at Cologix 3 (ex Canix 3). We have a few bays, two routers and a "small" 60Gbps to New York/Toronto/London. Interesting: Montreal/London does not go through the U.S. but directly from Canada.
This is the network that allows us to do the ALPHA and the BETA. So later we're on track to having an excellent connectivity to Montreal.
In parallel, we're working on setting a high capacity, high availability network between:
-BHS and New York (bypassing Montreal)
-BHS and Chicago (bypassing Montreal)
The goal is to reduce latency (so it does not go back to Montreal) and manage large bandwidth capabilities between our DC at BHS and two main POPs in the U.S.: New York, NY and Chicago, IL.
One of our added value is the network and quality of the bandwidth. We can ensure this service because we buy the dark fibre in Europe, we choose the equipment to run the fibre and we manage the network with our own teams.
Since we want to replicate in the U.S. the business model that has made us successful in Europe, it is indispensable to operate directly the optical fibres.
And it's not simple. Why? Because the existing operators do not really want to sell us the fibre and let us operate it ourselves. They
prefer to operate the fibre for us and sell managed services. There is surely a business model problem (and therefore the price) but mostly that approach prevents us from enjoying the technological innovations (such as Coherent 100G) and the same goes for our customers.
But culturally, the operators here do not sell or do not buy the fibre but exchange or swap at the ratio "1 metre against 1 metre". It is therefore necessary to invest in our own network (do some civil engineering, dig the ground) and after, only offer fibre exchanges with the companies that we're interested in, opposed to the companies that are interesting for other operators. Fortunately not everyone does all the exchanges...So what about this network?
We will sign in the coming days a fibre pair between Montreal and New York (via Albany) with a provider that is ok to discuss with us. This will allow us to turn on our own network with 100G coherent and connect our DC to New York with an initial capacity of 1Tbps. This network should be operational in August / September.
We talk in parallel with a provider for a second route (redundant) between Montreal and New York. This is a discussion which can result in 2 to 3 weeks for a network in place in September. The purpose is to have two routes and redundancy.
How do you have redundancy?
To ensure redundancy, one of the solutions is to take the 10G circuits on one of the routes and the 10G circuits on the other routes. Then connect everything on the routers. With the OSPF / BGP routing protocol, the router will use the route that is UP, and if it's DOWN, the router will recalculate the route to use.
The problem with this solution is that it works well on small networks of 50-100G but not in our case where we want 1Tbps for a start and then 8Tbps. To connect 1Tbps of real bandwidth means connecting 2Tbps of capacity. That means 200x 10G instead of 100x 10G. This is a very expensive solution since a 10G circuit and every 10G port on a router costs a lot of money. Above all, the router is quickly limited in number of ports per chassis. So that means several chassis of routers and in the end, a backbone that is very complex to manage.
And we want a simple and cheap backbone.
The solution that we would like to put in place consists in using 2 optic routes and to failover the fibre optic but not the IP on the routers. Level 1 but not at level 3. It means that the 100x 10G that are connected to the routers are always up and the optic system uses one optic route or the other. It is called "protected circuits" and with 100G coherent it becomes very affordable.
That's why working on our own fibre optics is necessary. Optical equipment on two routes must all be of the same brand and must communicate together. The aim to switch the optic route to <200ms if there is a failure.
In Montreal / New York we are discussing about a 3rd route with a provider that have fibre between the Canadian border and Albany, NY and another provider that has fibre between Albany, NY and New York, NY. A 3rd choice then. We need 2 providers to ensure 100% SLA, so that's good!
Let's continue about our network.
On the other side of Montreal, to Toronto, it is a bit more complicated, even if we have several ongoing discussions. The law requires that to become a provider in Canada, 80% must be owned by Canadians living in Canada. Therefore there are 4-5 big providers, low competition and therefore low new investment. The last one dates in 1999-2001. Since then the providers have been using fibres that are in place and keep them for themselves.
But we still manage to establish contact and to mover our project forward. First, on an optical fibre between Montreal and Toronto via Ottawa with a provider who is willing to offer it. There's no deadline yet because we are not interested in Toronto but in Chicago. Between Toronto and Chicago, IL there are two routes: - Via Buffalo, NY & - Via Detroit, MI and this is where things get very complicated. On the path between Toronto and Detroit, MI and Detroit, MI and Chicago, IL there are 3 operators who have fibre networks and this is the same between Buffalo, NY and Chicago, IL. But communications are complex and it's the same situation on the path between Toronto & Buffalo, NY. Why?
Because it happens on the paths between the 3 exchanges: New York, Toronto, Chicago, the paths are very short and direct and as a result, you get "low latency" and therefore the paths are used for HFT (High-Frequency Trading), trading at high speed. Servers passing orders on to each exchange speculate on the orders that another has previously sent. The goal is to process orders and jump the queue and enjoy the difference of the short exchanges. There are companies who are willing to pay M$ to gain a few ns (ns, not ms)and pass orders on the stock exchange before their competitors get a chance to speculate on the exchange. Therefore there is a complex negotiation of fibernoire on these paths because it is very expensive and the operators do not want new competitors (to continue making money). So if there are any problems, they are mostly from Toronto to the USA.
That is why we are discussing with several co-operators on the construction of a new network between via A20, instead of going through Ottawa. even if it is a project of 2 years it is very interesting because the network in question will be shorter and therefore faster.
And after New York and Chicago?
Once we are in New York and Chicago, it will be easier to establish a fiber network. Even if operators redeem themselves and market Fibrenoire (forcing them to offer managed services), there will still be operators whose primary business focus will continue to be Fibrenoire.
Why Fibre in Miami?
With 100G bandwidth, it is possible to put out an optical signal over 3500km without the need for regeneration. The current 10G technology (used in Europe) can travel only 700km. So what we want to do is generate a signal that will be able to travel from Miami to Montreal without regenerating. This will in turn reduce costs and latency. A real innovation for the world of Telecoms especially in the USA where distances from coast to coast can be up to 7500km. (the distance for fibre optics.) Regenerating the optical signal with 10G all 700km or regenerating against 10x 100G of the optical signal every 3500 km or 2x 10x is more bandwidth and 5x cheaper
And our network will have all this?
On the west side, the POPs we want that will reach large capacity bandwidth are Los Angeles, CA and Palo Alto, CA. On these two POPs, there will be operators in Asia (Japan, China, Singapore) and Australia. Now the fibre POP in Chicago, IL is connected directly to BHS. For the meantime it will be with 10G.
Otherwise?
Otherwise, we have tens of requests for 10G on different paths. The proposed 10G standard on which we built the managed network. It costs a lot but it is a map to the overall network that is interesting to have.
The finale is the network we want to have in the USA:
http://yfrog.com/z/h0l8uah
Just like that
That's it. Make the network as it has never before. Simple? If it were easy, everyone would be doing it. If it was simple, there would
no added value. If there were not challenges to this project, we would have not started it. In short, we sought out the difficult challenge.
We will see in six months.
Regards,
Octave
