Update -- 9/19/2002 - problems with packet loss appear to have been fixed; posts below are for reference sake in the event of future outages with Verizon's CDPD or Express Network/1XRTT services. There still is a good deal of latency on the 1XRTT system, ie, ping return times of over 400 milliseconds, but this is more or less the same (not too great) level of service it has always been at. -- Hi, Earlier today (9/17/2002) I posted about Verizon's Data Services having very large amounts of packet loss, even after their outage of last week was supposedly fixed. I just got off the phone with a gentleman named "Omar" at the Data NOC East, and he said they were doing some testing in the New England area tonight which may account for the problems. However, the problem is also occurring in the NY Metro (00022) market to an extent and has been going on in the New England markets (Wallingford/00119, Manchester/00428, and N. New England/00028) since this afternoon, so I'm not sure it purely a late-night testing issue and am beginning to think there is something more systemic going on which is debilitating customers' use of Verizon's data networks. Right now, on most of our CDPD IPs were are getting between 20-30% packet loss, and 15% packet loss or greater on the 1XRTT system. Due to the current extent of the packet loss throughout the New England (and probably other markets), both services are nearly unuseable! :( Omar suggested that if the problem continues tomorrow (9/18/2002), to call into the Data NOC and open a trouble ticket, which, at this point, after a week of problems (today being one of the more severe days), I intend to do immediately if the problem is still going on. If anyone in any other Verizon market is experiencing similar packet loss problems, please let me know so I can add your observations/experiences should I open up a formal ticket on Weds. Thanks! (This post and SID list are also available at http://www.wirelessnotes.org) Regards, Doug Interpage(TM) Network Services Inc. / http://www.interpage.net ----- 3PM Eastern, 9/17/2002: Hello, After last week's outages on the East Coast (especially in the New England area), it seemed like BAMS/Verizon's CDPD and 1XRTT Express Network problems had been resolved. However, just today I did tests with a Jersey City-based CDPD IP, a Woburn CDPD IP, and our only 1XRTT account (dynamic IP), all of which were showing packet loss in the range of 20 to 30%. The JC IP was located in Madison, CT, right at the rest area NB on I-95, and was stationary. There was a -60dB signal, or better. Packet loss was generally around 22% at 3PM, Eastern, 9/17/2002 The Woburn IP was stationary in Milton, MA, on MA-28 near the Milton Academy. Generally OK CDPD service there, around -80dB. Packet loss was 30%, again, around 3PM. The 1XRTT (what I'm using now) was also located in Madison, CT., and experienced about 10-15% packet loss, from 2 PM till about 5PM, and now it seems a bit better (only 3% or so packet loss). We've had CDPD accounts for a number of years now, with both BAMS and AT&T, and rarely have much if any packet loss (*latency*, yes, but in good or decent coverage areas little to no packet loss), which is why I find this unusual. Anyone else notice this or have any idea what is going on? (This post and SID list are also available at http://www.wirelessnotes.org) Regards, Doug Interpage(TM) Network Services Inc. / http://www.interpage.net d1@interpage.net ---- The following were posted on or about 9/12/2002, which do not relate (I think) to the above post, which deals with a more generalized problem (packet loss) on BAMS/Verizon's network as compared to the specific outage/issues detailed below: ---- ---- Combined posts from 9/10/2002 regarding Verizon/BAMS Data (1XRTT and CDPD) problems: We've been using a few Verizon Express Network accounts (with Sierra 555 cards) since June after they implemented the Unlimited/$99 per month plan. Despite some billing mistakes (wrong rateplan) early on, we were generally pleased with the service. When we first started out, 1XRTT seemed to offer some noticeable speed improvements over CDPD, mainly that we got effectively a 40KBps wireless modem at all times (where there was coverage, of course) with bursts which approached 120,000. However, over the past few months, and more even more so recently, we've noticed: 1. At late hours local time (like 2-4AM),. a connection will just drop for no reason. We could be FTPing a file or actively telnetted on somewhere, and for no reason the connection just drops and we have to reconnect. Frequently, we can't reconnect for like 30 minutes or so, and after that it works fine again. (As if they are working on the system and don't want people using it then.) This happens in many markets, from Maine to VT to Philly, DC, etc. If this happens, not only is work interrupted, but the entire file has to be re-FTD'd , etc., which wastes time and is generally a poor showing for such a "premium" service. Additionally there are certain spots which when mobile always result in a drop, regardless of what time of time, regardless of how "good" the coverage ie, such as the Saw Mill River Parkway in NY just south of JCT I-287, the Outerbridge Crossing on Staten Island (NY/NJ-440), US-1 in Kingston, RI, etc. When you drive though these areas, the 1XRTT will go from "connected" to "ready to connect" and you need to re-establish your session all over. This is NOT due to a lack of coverage, it just seems to do this for other reasons. 2. The latency has gone from acceptable to worse -- we are getting 400 - 700 ms delays during the day, and about 350 to 500 ms delays at night. I've compared this to CDPD accounts which we have with Verizon/BAMS as well as AT&TWS and they are frequently LESS; BAMS CDPD with a Jersey City IP to our UUNET hub (T-1) is about 300 ms day and 110 night; AT&T to our Qwest DSL site in Boston is like 350 ms day and 250 night. (These are all very rough averages.) I called them a few times about this and they never have a decent answer other than "a lot of people are using it; we'll look into it some more". 3. There are days, like today, when there is even some degree of packet loss (something you rarely see with CDPD), which makes any attempt to use the service in a manner similar to a 56K connection (which is, in part, how it is promoted) impossible. I've had to disconnect and re-establish from my host system a number of times today, and I'm in a known location where 1XRTT (as well as CDPD service) has always been excellent. Indeed, I got so fed up with the 1XRTT I yanked out the Sierra 555 card and put in the 300 CDPD card and am typing this now using CDPD since it is actually a lot better! :( 4. Frequently, the speed is not much better than CDPD; we can get on a good day 5 or 6 Kbps with CDPD (-60 or so dB level); in many areas, the 1XRTT is not much better and on occasion slower than that. 5. CDPD seems like a much more robust protocol -- if you don't pass data for an hour and then continue with your telnet session, it works just fine; if you drop out of coverage and come back an hour later data which was stopped generally resumes, etc. I realize CDPD was designed with public safety and the like in mind, but if 1XRTT and similar, higher-ended services will eventually replace it they need to bring it up to the robust level which CDPD is at now. Currently, it would likely not be an acceptable alternative for public safety and other mission-critical applications which require wireless data. Has anyone else noticed these sort of problems?? I've tried it from a variety of areas in New England and the Mid-Atlantic region, and have obtained similar results in all these areas, especially in terms of latency, which is approaching satellite levels and is unacceptable. Overall, I like the speed (when it works), but the latency, lack of reliability, disconnects, and other problems above do not make it a suitable alternative to CDPD (yet) and for what we are paying it is probably not worth it if they don't fix these problems shortly. Regards, -Doug d1@interpage.net www.interpage.net / www.wirelessnotes.org ------ Hi, Anyone notice there are AWFUL latency problems going on with Verizon/BAMS wireless data services today (Tues, 9/10/2002)? All of our Wallingford and New England CDPD IPs and Express Network 1XRTT accounts have been showing terrible latency. Ping ms times are not terribly worse than normal, but data will go back and forth just fine for a while, then everything will stop for like 30 seconds ("freeze up" is the best way I can articulate it), then catch up, then stop, and so on. This makes it impossible to do anything interactively! Anyone else notice this? It is not the usual "busy due to daytime usage" occasional one or two second delay... A guy who works for me in RI said he was doing a ping and it did 70 or so pings just fine (1 ping per second), then it just stopped for 30 seconds, and then went on again and worked fine for a minute, stopped, etc. There was NO packet loss, just this "freezing" of the connection. The employee called Verizon's Data NOC and after initially being more or less told "Well, we can't be responsible for latency in the sites you need to connect to" (we use them all the time and connect to many sites and it is never like this; if it was typical we'd have dropped them long ago) was told that they were having problems in the New England area and although they were not considering it an outage they were trying to see what was going on, ie, was it just too much traffic or something else...(Too much traffic? At 11PM?!?!?!?) Anyhow, if anyone else has noticed unusual problems recently, please let me know and mention your market and/or where you were when the problems took place... Regards, -Doug d1@interpage.net www.interpage.net (This post and SID list are also available at http://www.wirelessnotes.org)