Talk is cheap... well, cheaper…

UW long distance rates have long been higher than industry rates.  However, according to a memo from IST:

“Effective June 1, charges for long distance calls made from campus will be 2 cents per minute for all calls within North America.”

International calls will be billed according to certain rates.  Unfortunately for Engineering the United Arab Emirates calls are billed at 95 cents per minute.

Hopefully these new rates will reduce the administrative effort dealing with personal calls, which would often add up quickly under the old rates.

Questions: ext. 36609.

These prices greatly reduce the need for people to use Internet based phone technologies like google voice or skype.

Guess who wants your password

If you’ve been on campus more than a few months, you’ve probably received some phishing attack where the bad guy tries to convince you to send him your userid, password, and maybe more like your birthdate, often under the guise of cancelling your UWaterloo computer account if you don’t.

For the record, you will never be asked to submit your password in an Email.  IT staff don’t need it for most things you ask them to do.  And if we do need it, perhaps to debug a problem you are having, one of us would be sitting at the computer with you.

But what, you may ask, does my password give bad guys?

Overwhelmingly, it supplies them with an account from a trusted system that can rapidly send their spam.  We throttle it back after a few minutes, but they can get a good number of free spams off, and someone will read that spam and buy some product from them, and they will make a bit of money.

Another gem on your account is the list of Email correspondents.  They have a list of people they can Email, and probably those people will read that Email if it comes from your account.

If you are like many people, your password also works on Ebay, PayPal, FaceBook, etc.  There they can clean out your accounts or advertise a product.  In particular, FaceBook lets them target your friends with ads. 

At UW, like many institutions, we have a Virtual Private Network or VPN which can be used to grant any remote machine network access as though it were physically located on campus.  Your userid/password pair is enough to gain access.   The VPN lets you do dastardly things on campus.

One of the growing threats is impersonation of campus people in order to steal online resources like online journals.  The University spends 4.5 million per year on electronic journals.  Your userid may be sold on sites just for your library access.  It may surprise you that there is a world market for this.

Your birthday can be enough to get access to other resources.   And also the bad guys can use your payroll information (accessible here with your password) to get your social insurance number.  This can be used for full scale identity theft.

Many people store important information on computers, either directly in files, or unknowingly in their browser cache.  If someone were to snoop, he might find bank accounts and other data, however this is not yet as common as some of the other abuses.

Today many homeowner insurance policies will include identity theft protection.   It actually is a real risk in this age.  It won’t remove the frustration you feel, but it greatly reduces the risk of losing all you have worked for over the years.

Have safe computing.

Erick

Engineering Computing File Server

In December 2010, Engineering Computing moved to a new file server.    

ECFile1 is now a 64 bit, dual quad core computer with 72 GB of RAM, running FreeBSD with Sun’s ZFS file system and serving CIFS/SMB (using Samba) to primarily Windows clients and NFS to Unix servers.

ZFS offers a number of advantages over other file systems like BSD’s UFS or Linux’s EFS.  Thanks to its inclusion in the OpenSolaris project (which has been shut down since), the open source community had access to the complete source code to this revolutionary file system.

ZFS allows almost unlimited growth – well, up to 16 exabytes or 2^64 bytes per file.  That’s about  100,000,000,000,000,000,000 bytes.

ZFS also focuses on data integrity with technology to detect problems better than standard RAID, and to actually heal the data.  It actually works better with non-hardware-based RAID, where ZFS can test the data from multiple disks against a pre-computed checksum to determine the most reasonable results.

Modern disk drives use approximately 25% of their disk surfaces for error detection codes.  Enterprise SAS drives have an undetected error rate of one in every 10^16 bits.  Without the advanced coding of ZFS, enterprise systems were experiencing systems reliability and data errors due to the disk error rates.

Another great feature of ZFS is the ability to use solid state devices as intermediary devices between the disk drives and the computer to speed up both reads and writes – significantly reducing the load on the physical disks while gaining the speed advantage.  Writes are typically buffered for about 30 seconds, and then flushed to disk at once.  ECfile1 uses two SSDs.

Currently ECFile1 has 36 SAS disks, some are hot spares.  We have 12 terabytes of space dedicated to user storage, but only 20% of it is in use for user data, and some additional space is used for snapshots.   When we put it in place, we doubled most users’ disk quotas.  We should be able to increase them again.

An advantage of a copy-on-write system like ZFS is that old data can be retained.  This allows us to have snapshots – allowing us to recover files from previous states.  When doing backups, we take a snapshot of the system (it takes a second or two), and then back up the snapshot.

 ECFile1 is connected through a 10 gigabit per second connection to the Engineering network.

Backups go to a second similar box (but with SATA drives) in a different building.  In the event of a catastrophic failure (like a fire taking out all the disk drives on ECFile1), the other box could easily be brought up as ECFile1 with only a few hours of data loss.  

More frequent snapshots could move the catastrophic data loss down to an hour or so.  And work on clustering might eliminate it entirely.  These are all options for the future.

For now, we have a fast, reliable file server.

Reliability : Uptime and Downtime

Downtime

Downtime (computers or networks being unavailable) is one of the most frustrating experiences as a user, and something IT staff work tirelessly to avoid.

Downtime can be caused by hardware or software failures, security breaches, or as a planned exercise so we can change hardware or update software.

We try to avoid downed systems by using stable software versions, more expensive and redundant hardware (multiple network cards, multiple fibers, RAID disks, hot swap hardware, etc.).

Most of our systems are rebooted only to install operating system security updates, and we do that in the middle of the night.

Some hardware upgrades are unavoidable, but we also use the opportunity of power outages when we can.

Our schedule is complicated by the need to be up while students work well into the night and during class time for the Dubai campus too.

In the computing industry (and engineering in general), uptime/downtime is expressed in nines.

Nines	Availability	%	Downtime per year
one nine	0.9	90%	36.5 days
two nines	0.99	99%	3.65 days
three nines	0.999	99.9%	8.76 hours
four nines	0.9999	99.99%	52.56 minutes
five nines	0.99999	99.999%	5.26 minutes
six nines	0.999999	99.9999%	31.5 seconds

So how do we fare?   Many of the background servers are rebooted for security patches once or twice a year, they offer five nines.  Since rebooting takes between one and three minutes, five nines is the best that can be achieved without resorting to totally redundant servers.

The Engineering file server was in that category for four years, but last year it started having problems in the Fall.  Its replacement in December 2010 has been totally stable again – expected to be five nines again.

The engineering network backbone  (for CPH, E1, E2,E3) has also been five nines for a while.  The login web browser on the nexus login screen depends on a web server that was at four nines last year – we’ve since moved it.

The campus homepage web server was down for a while on the snow day.  It doesn’t take much downtime to degrade your nines.

The two network resources with bad uptimes this term were wireless and the campus external Internet connection.  Both are managed by IST.

Wireless is always less reliable as it is subject to interference.  However, there is more involved than that.  IST uses a SandVine installation to shape wireless traffic and reduce peer-to-peer traffic.  That had problems.  Also there were unexplained reboots of wireless access points and other oddness.   With scheduled and unscheduled outages and configuration problems, wireless operations were  definitely problematic this term.  I don’t have enough information to state the nines, but I wouldn’t be surprised if it was around two for some areas of campus.

The other problem this term was our off-campus or external network links.  There were a number of hardware and fibre related problems between us and Toronto.  Several times we had partial outages where only a percentage of the network traffic was lost, sometimes only between certain computers and not others.  We have a third network provider and our systems are supposed to fail over to the redundant link.  This takes some time to take effect, so it’s not immediate.  Also, our redundant link was not as fast, leading to congestion.

IST is planning to improve the external links.  It’s a good plan, people are depending on our external connections.

It’s often said that you only notice your IT staff (computing staff) when things go wrong.   We’re trying our hardest to make that not happen.

Computers Can Do That

In Engineering Computing, much of what we do is related to automation – making computers do the work that otherwise people are forced to do.  An obvious example is Nexus, where software is automatically distributed to workstations, but there are others.

We are involved in several database-related projects which replace paper-centric processes.  The biggest of these projects is OFIS (Online Faculty Information System). 

OFIS has information about all faculty members, grants, research, publications, appointments, etc.  It is used for yearly evaluations of faculty members, CEAB accreditation submissions, managing sabbaticals, and much more.  

On the student front, we have been doing web based PDEng surveys for a few terms, and now that has expanded to some graduate courses critiques.  This removes the paper process and reduces effort by the Dean’s office.

In many ways, UW relies on paper or scanned images of paper.  Probably later this year UW will acquire a certificate server and implement a PKI public key infrastructure.  Then it will be possible to electronically sign documents and avoid having to print/sign/scan as we do today.

The goal of automation is not to reduce the number of employees.  The Faculty of Engineering is in an exciting growth phase; automation will make people’s jobs easier, and may reduce the number of new people we have to hire.  Computers should reduce the repetitious work and increase efficiency.  That is our goal.

Social Media and You

Many readers of this blog are students, they are among the Internet’s biggest consumers and producers of social media content.  They are more connected than any previous generation – with cell phones and other devices keeping them online and interacting constantly.  This has created many opportunities, but has also lead to some new forms of bullying, stalking and other crimes.

Some of you blog, tweet, or at least update your status on Facebook.  Many companies have rules about what can be blogged, sometimes they are quite permissive, other places are very strict.  Be sure to learn about those policies when you are employed.

Schools also have rules of conduct.  More than one student has been surprised to find there can be academic consequences for their actions.  Everyone at UW is governed by Policy 33.  I recommend you read it, it’s good to know what is expected of you, and whom you can contact if you or a friend needs help.

http://secretariat.uwaterloo.ca/Policies/policy33.htm

If you ever experience harassment or other crimes while on campus or the Internet, get to safety and then get help.  I would be glad to assist if your crisis involves computing or networks. 

Erick@uwaterloo.ca

Wifi in classrooms

UW Wifi

The UW campus wireless system is built with technology from Aruba Networks.  There are approximately 1,000 Access Points (APs) in the main buildings and 600 APs in housing locations.

Most campus building use 802.11 b/g, whereas new buildings (e.g. E5) use a/b/g/n, which allows for higher speeds and more channels which can be used with more APs to address more concurrent clients.

Usually each AP can service 10 people well, up to a practical maximum of 20 concurrent users before service becomes too degraded.  To plan the numbers and placement of APs, the intended use of space is considered.

For low density usage, such as offices, 1 AP per 6,000 square foot is about the minimum density and was the standard density used for most of campus as Wifi was not initially used extensively.  To achieve higher density of clients, APs can be moved closer together.  APs function best when each has about 2,000 or more square feet of coverage.

Larger teaching classrooms can hold 120 students or more.  In most cases, we cannot accommodate everyone using a laptop concurrently with the existing deployment.  To make the rooms usable requires multiple APs, one for each group of 20 users packed in 300 square feet.  The approximate cost is $50 per seat assuming 20 users per AP at $1000 for conduit, cabling, AP, etc.

IST is responsible for Wifi.  They know coverage is inadequate in numerous locations based on complaints from users.  You can leave messages about wireless at http://ist.uwaterloo.ca/request

Exciting new technologies using Wifi will help us engage students in the coming years.  That is not fully possible with the current coverage.  But, of course, until the facilities exist it is frustrating to impossible to try these technologies, so they must be a priority.

As always, Email me if you have comments or questions.  erick@uwaterloo.ca

Recent Lab Upgrades

With the backing of our Dean, Engineering Computing renews our nexus labs every few years. 

Last year we added the new Gear lab and the Student Design Centre. 

Last year the WEEF lab was overdue for an upgrade; it was completed by the start of the Fall term.

In January, 2011, an upgraded Helix lab came back online.

We're working to move some labs to Windows 7, not all the software is compatible yet, but we'll get there soon.

Let me know how we’re doing.

Friendly Email Address

The account name you log into when you use Nexus, check Email, or use Ace is usually initial then surname all truncated to eight characters.  And if you are unlucky enough to have a popular surname, you may have a number inserted to make your userid unique, and then cut off at 8 characters.  I count 2,613 people with the surname ‘Wang’ currently registered.

Maybe you would like a more personal Email address, one that doesn’t have a number and doesn’t truncate your last name.  Well, they are available to all members of the UW community.

A feature called friendly Email addresses lets you get Email addresses like Steven.Brown@uwaterloo.ca or Steven.M.Brown@uwaterloo.ca, etc.  It forward Email from this friendly address to your real Email account.  You can tell your mail program to specify the friendly address when you send mail, and users will come to know you only by the friendly address.

To learn more about friendly Email addresses, see http://ist.uwaterloo.ca/cs/FriendlyEmailInvitation.pdf

Erick

Moving Email to IST

This post was published in the February 2nd Iron Warrior.

The Faculty of Engineering is getting out of the Email business.  Student Email is moving to IST’s Mailserivces during reading week while faculty and staff are moving to IST’s Connect over the coming months.  Students who use MyWaterloo.ca and let the system auto-select the Email server will be redirected when the accounts move to Mailservices.  Your @engmail address will be valid for quite a long time, we will forward Email to the new addresses.  Your preferred new Email address will be userid.@uwaterloo.ca

How things change.  During the entire month of October1988, Engineering Computing processed approximately 1,000 messages to our undergraduate network from the Internet.  Within two years, we were processing 1,000 messages per day and deploying new systems to keep up.  Metcalfe's law states that the value of a telecommunications network is proportional to the square of the number of connected users of the system.  Our Email growth reflected the rising connectedness of the world.

The first mass Internet spam was in 1994.  Spam quickly became a parasite on servers, and attachments added load, forcing us to upgrade to ever more powerful servers every few years.  Now it is pretty common for 1,000 message to blast to our Email systems per minute (or more if they could), and the vast majority of it is spam, much of which is discarded before you see it.  Not only are the spam rates climbing, the spammers are getting more sophisticated and harder to detect.

Today our Engineering users communicate with people worldwide and they themselves are offsite – 65% forward Email to Gmail alone.  This makes sense; Gmail does a better job than we can ever do on campus.  They offer more storage, a better user interface, better spam handling, Email for life, etc.  The days when UW could compete on Email are long over.

For the last few years, Engmail used an open source package called SpamAssassin to try to classify spam using a variety of techniques. It does not catch all spam (I’m sure you noticed this), and it has false positives which may cause some valid Email to be treated as spam.  IST also uses SpamAssassin for Mailservices and Connect.

For added measure, IST use gray listing, which uses a system of delays to try to weed out spam.  Under gray listing, a good percentage of off-campus-originated Email is delivered hours later than it was sent.  Under testing Engineering Computing staff found these delays rendered Email effectively useless.  Worse, several important Emails never made it through.

IST’s gray listing is already turned on as part of the transition.  You can turn off gray listing for your Email account at  http://mailservices.uwaterloo.ca/greylisting-toggle.php

Many are questioning why UW is still offering Email locally.  Google does a far better job, it’s also a lot cheaper and the money saved would be better spent on other endeavors.  IST’s mail clusters consist of more than 20 server class computers; it’s expensive and history suggests it will need constant staffing as well as frequent upgrades and overhauls.

Youth today are relatively unimpressed with Email; they favour other technologies for most of their communications. There are essentially two reasons why people feel students need UW-related accounts (though they could be hosted offsite).  Students need a way to communicate with faculty that is secure and uniquely identifies them (UWuserid) – but Email is still an insecure protocol, anyone can impersonate anyone else - a closed messaging system on Ace would be better.  The other factor is that students prefer to correspond with employers and potential employers with a UW branded address.

Google Apps for education would offer Email for life, calendaring, 7GB of storage, word processing, spreadsheets, sophisticated sharing, all with no ads and excellent privacy. It’s already used by thousands of universities.  Microsoft’s Live@Edu competes with a similar array of features and followers.  

However, there is an impediment to transferring student (and faculty and staff) Email to Google or Microsoft – neither company will guarantee that the data will stay in Canada and be subject to Canada’s and Ontario’s privacy laws.  Most likely it would be hosted in the United States and be accessible under the Patriot act.  WLU got around this problem by suggesting Gmail accounts (with WLU branding), but offering optional WLU locally-hosted Email accounts for anyone who preferred the local account.  100% of the students took the Gmail option.

Graylisting - Why is some Email taking so long?

Effective January 2011, Email to all Engineering students (and many faculty and staff) has been going through IST's mail cluster.  There is an article in the February 2nd 2011 Iron Warrior explaining that in more detail.

The cluster uses a system called graylising to try to reduce spam.  Graylisting uses a strategy of delaying Email to see if the sender is probably more legitiamte than many primative spammers.

In my experience, Graylisting has some limited success but at the cost of timeliness.  Email from an offsite address that hasn't written to you in the last seven days will often be delayed, either minutes, hours, days, or sometimes the mail will be lost due to graylisting.

Friends may become upset they can't reach you, or may make lunch dates that you don't know about.  Several people in Engineering Computing ran into problems when we were first subjected to graylisting.

Graylisting doesn't approach the level of spam control on gmail. 

If you expect Email to be delivered in seconds, you will want graylisting turned off for your account.  You can do that by visiting: http://mailservices.uwaterloo.ca/greylisting-toggle.php

Erick

Putting in the Hours

Some Engineering plans use nexus more heavily than others. 

Here are the median hours spent by people in each department and term during fall 2010.

The chemical numbers are lower than the true values because ChemEng wasn't running the software logging usage on their lab computers, so time spent in ChemEng rooms were not coujnted.

If you have any comments, feel free to Email me at erick@uwaterloo.ca

Giving Feedback

As I write this blog entry, Tunisia, Yemen, Egypt and Jordan are going through major political change.  By the time you read this, I’m sure the status of these countries and probably others will have changed again.  It’s an incredible time.

In most of these recent events, the Internet has played a strong role; WikiLeaks was partly responsible for Tunisia’s demonstrations and services like FaceBook and Twitter helped to fuel the flames and organize protests.

In Canada this year, FaceBook and Email campaigns have had impacts on decisions at various levels of government right down to the uwaterloo logo, lasers anyone?  Or Engineering’s PDEng program – our faculty now conducts student surveys on PDEng every term.

What many people don’t realize is that Canadian governments and institutions like UW actually want your input and are increasingly embracing the Internet to collect your views.  As citizens we often think our only power is to turf the players out every few years and complain the rest of the time, but we can also write email, join facebook groups and participate in online surveys to help get change more quickly.

Savanna, our co-op this term, is conducting a survey regarding computing in the faculty.  You can help us plan for the future by completing the survey online.  It’s anonymous and only takes a few minutes.

http://www.eng.uwaterloo.ca/computingsurvey/

Feel free to write to me at erick@uwaterloo.ca if you have comments regarding computing in Engineering or at UW, good or bad.

NOTE The survey is now closed.  Savannah will summerize the results shortly. Thanks to the hundred people who participated.

Printing Costs Starting January 2011

The charging rate for printing increased this month, going from 8 cents to 10 cents per 8.5x11 page B&W, and 25 cents to 50 cents per 8.5x11 page colour.

The new prices are campus-wide and for Engineering are the first price increase in many years.

Engineering Computing manages 15 printers in our labs and some departmental labs, some offices, and the colour printer located near the help desk.  We print hundreds of thousands of pages per year, though the number has been steadily declining as more work can be submitted electronically.  Last year students printed 400,000 B&W pages on our printers.

The price of higher end printers has actually been falling, but so has their quality of construction; newer printers may require more frequent replacement.  Also, the cost of consumables (paper, toner), maintenance and co-op labour (to refill the printers) are all increasing.  WatCard’s 0.5 cent per page transfer fee cuts into the price too.  But the biggest factor is that we are now hit with GST and PST, whereas we were previously exempt.  That 13% increase was much too big to absorb.

The cost of running the printers is higher in Engineering than other faculties because we maintain so many printers in all the labs.  It is more cost effective to have two or three heavily used printers and force everyone to walk to them, however, that would not be as convenient for students.

About 13.5% of student B&W jobs are printed duplex (double sided), leading to 15% of our printed pages to be double sided.  Duplex printing is charged at a rate of approximately 75% of printing two single sided pages.  The actual cost to us of printing double sided is closer to 85% of the rate of single sided.

In 2009/2010 we printed 20,000 colour pages.  We never hoped to break even on colour printing, but last year we were losing too much per page at too high a volume to continue at these rates.  Other faculties were charging several times our rates, and the CTSC committee decided to standardize on a new fair rate for everyone.

With the new rates we can afford to replace printers sooner. We hope that will lead to fewer breakdowns and clearer output.

Printing at UW is still a good deal.  B&W output on a home inkjet costs roughly 50 cents per page.  Colour output at home is usually even higher, though it depends on how much ink is used.

If you think a particular printer is giving poor output, be sure to contact our consulting office so we know to have the printer serviced.  We are alerted when it runs out of toner, or when it is running out of paper, but we can’t tell remotely if it needs a cleaning or other work.

Introduction

Hi, I'm Erick Engelke, Director of Engineering Computing at the University of Waterloo. 

My department takes care of many (but not all) computing issues within the Faculty of Engineering, and I also sit on the university committee (CTSC) which makes decisions about all computing-related issues for the campus.  I cannot change everything, but I do have input into the process.

I've decided to start blogging to share information with our clients, and to offer a forum for feedback.  Feel free to post or to Email me.

To students, we are best know for Nexus, the computing system that delivers software to approximately 1000 student lab computers in Engineering (approximately 500 of them in Engineering Computing labs, 500 in departmental labs), and about 4,400 computers across campus.

In the last few years, students' own laptops have added to the picture, but not in the way many people expected.  Laptops are useful for many things, and they reduce the reliance on labs for typical commercial software such as Microsoft Office.  However, over 95% of our undergraduates still use nexus labs and our Terminal Server (remote desktop to Windows).

This table is based on the fall 2010 term.  The co-ops, Masters and PhD students sometimes use private office computers, so they are not necessarily adding to the lab traffic.

The labs are often busy, especially as we get closer to end of term.  By November, it's hard to get a spot anytime between 11am and 6pm, or even later in Gear. 

Usage is higher in certain labs, mostly where we have newer computers, or more workspace.  Last term the gear lab was used heavily until 11pm most nights. 

There are several reasons people use labs in addition to laptops, but one of the most prominent is that the labs have specialty engineering software installed.  I've heard other reasons including monitor size/screen resolution, high speed network connection, not lugging a laptop, dead laptop batteries, working in groups for projects, working near friends, etc.

Some students have discovered Engterm, our Windows Terminal Server.  It has many scientific applications on it, but not all the ones in the student lab - some licences prevent us from using software on a terminal server.

One of the benefits of Engterm is that you can access maltab, maple, mathcad or certain other packages from your laptop, at home or school, without having to find a spot in our labs.  To use it, find remote desktop on your computer, and type in engterm.uwaterloo.ca.

Engterm usage seems to peak between 8pm and 10pm, probably when people are working on homework at home.  The graph misrepresents low usage in the early morning hours, as the counters are reset at midnight and ignore people already logged on.

As you can imagine, there are many issues running big computing systems, computing networks, etc.  I have a list of potential topics for this blog but questions from users will probably shift the priority.

I hope your find our systems useful.
Erick