OpenWetWare:Infrastructure

From OpenWetWare
Jump to navigationJump to search

Ideas for keeping OpenWetWare running:

  • Drew + iCampus funding is available to get off-campus hosting.
  • At Wikimania, Wikipedia is considering using PlanetLab (codeen.org). This is a global caching system with hundreds of servers around the world. MIT is already part of it, so should be easy to join. This doesn't solve reliability, just bandwidth and scaling.
  • Also Coral cache - content distribution network
  • Levels of outsourcing IT
    • Co-locating a server at a professional web hosting company would go a long way toward increasing reliability of the site. These companies have physically secure datacenters with backup power and redundant Internet connections. This kind of service would cost $150-200 a month.
    • Dedicated and Managed hosting

Check list

  • redundant power (different substations, UPS, generators)
  • redundant network (number of connections, physically separate)
  • DNS servers
  • Backups (amount space, frequency, time to restore, access to individual files, location, retention period)
  • support (phone or email)
    • Hardware replacement policy
    • OS support
    • Application support (compiling and supporting 3rd party apps)
  • Bandwidth/traffic

MIT

DOST: Data Center Operations Services Team] procide co-location services and server management; contact: 617.253.7049, dost at mit.edu

  • Datacenter in W91 runs attended service 24 hours a day with the exceptions: Saturday night, Sunday night, MIT Institute holidays, and MIT Institute special closings. Runs undergraduate student applications site, SAP applications, etc.
  • Co-location services: $600/year for up to 2U
  • Server management by ASST (Administrative Server Service Team); Harold Pakulat, 3-7728, manager of server operations group; Jody Housman, 3-3238, ASST
  • Services and Pricing: 1 point = $1,250/year, billed quarterly
    • 2U = 1pt
    • Supported OS = 1pt
    • 24x7 service (monitoring and simple recovery) = 2pts
      • 4-hour hardware replacement (3rd party company); if part is on site than less time
      • Support may not be available on weekends(?)
    • TSM backup = 1pt
    • Server cost (HP DL380): ~$2,000/year
    • Total: 5pts x $1,250 = $6,250/year plus ~$2,000/year: ~$8,250/year
  • Options for high availability
    • 2nd server (bldg E40) cost: ~$2,000/year for hardware (identical for easy maintenance) and ~4pts ($5,000/year) for support
    • 100GB of external high end storage (Symmetrix storage arrays synchronized via SAN using EMC SRDF) = 1pt
    • SAN is always synched to two locations, failover achieved by switching IP of the backup machine

ibiblio

  • Sri mentioned that ibiblio is a reliable web host who would be willing to provide us their services (dedicated hosting?) for free.
  • as a baseline ibiblio provides the basic services that any webhost would provide, including full access to your files and database, installing php and mysql applications, etc.
  • we serve our collections (websites) from clusters of webserver, with the data on centralized filers and databases. This architecture has existing backup in place and is the default location for essentially all of our collections.
  • access to Internet2

Commercial providers

Rackspace

  • Services:
    • Managed Service Level Agreement
    • Internal network connected to multiple ISPs at disparate access points; datacenter located in Grapevine, TX (near Dallas); 1000GB of outgoing traffic (promotional offer), incoming traffic not counted (a nice benefit, since wiki is not a read-only website), info about billing for bandwidth.
    • redundant electrical grid connections, UPS, can run indefinitely on diesel
    • SAS70 Type II certified
  • Sample server configs: hardware replacement time is 1 hour; load balancer, hardware firewall
  • Software
  • Backups: 25GB, full backup weekly, incremental backups daily; Legato tape back system, located in the same datacenter, retention: 2 weeks
  • Customer portal: DNS Administration, bandwidth and backup utilization, etc
  • Level 3 technicians on call 24x7x365
  • 100% infrastructure availability guarantee backed by SLA
  • Low-level intrusion detection system
  • Hosted sites include: theonion.com, nfl.com, pfizer.com, atari.com, etc.
  • Dell PowerEdge 2850: dual Xeon 3.2GHz, 2GB RAM, 2x SCA hot-swappable hardware RAID-1
  • RackWatch Platinum included: email notification, technicians will automatically fix the problem according to policy.
  • High availability using SAN
    • Cost would double for having it in two physically different locations

Data Return LLC

  • MHOne - utility computing solution based on VMWare Infrastructure
  • Company founded in 1996
  • SAS 70 (Type II) datacenter
  • Dual-path, dual-entry fiber facilities
  • Load balancer
  • Bandwidth: 1Mb/s(?)
  • Daily incremental, weekly full backup: 10GB. Two weeks of tape media stored on site, two more weeks are stored in an offsite facility.
  • Infiniserver: 2 virtual processors, 2GB "virtual" memory
  • Storage: 50GB
  • RHEL ES, 1 yr std support per server
  • MySQL - customer provided
  • Service level: MHOne Infiniserver
  • Pricing:
    • Implementation services: $750
    • Monthly services: $1,155
    • Software (RHEL, 1 yr): $799

Amazon Web Services

  • Amazon Elastic Compute Cloud (Amazon EC2) - in Limited Beta (August 2006)
  • Technical documentation
  • enables you to increase or decrease capacity within minutes
  • root access to each instance, and you can interact with them as you would any machine
  • each instance predictably provides the equivalent of a system with a 1.7Ghz Xeon CPU, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth.
  • Pricing
    • $0.10 per instance-hour consumed (or part of an hour consumed).
    • $0.20 per GB of data transferred outside of Amazon (i.e., Internet traffic).
    • $0.15 per GB-Month of Amazon S3 storage used for your images (charged by Amazon S3).
  • Fedora Core 3 and 4 are explicitly supported, but any distro based on the 2.6 kernel should work.
  • Slashdot article
  • Maluke Co. blog entry

NetNation

  • Infrastructure
  • Based in Canada, datacenter in Austin, TX
  • 1-hour hardware replacement
  • 100% network and power guarantee SLA
  • OS updates included but not custom application support (billable at $75/hour)
  • Backups: $50GB FTP space included, managed backups: $99/month
  • Dell PowerEdge 2850 with RAID: $450/mo, $350 setup (link)
    • Dual XEON 3.0 Ghz
    • 2048 MB DDR2 RAM
    • 3 x 73 GB SCSI (RAID 5)
    • 500 GB Bandwidth
  • less hand holding, support is billable at $75/half hour
  • Bandwidth: 500GB incoming and outgoing

C I Host

  • Dedicated Linux server: dual Xeon, 1.5 GB RAM, 2x80GB SATA RAID-1, $600/mo, $300 setup (with a 1-year contract)
  • 30 minute hardware replacement, 3,500GB bandwidth
  • only minimal support is included, extra support is billed hourly

The Planet

CSC - Computer Sciences Corporation

  • Based in California
  • quote pending

Logicworks

  • "enterprise level managed hosting provider": usually 10 servers and up
  • Basic High-Availability Solution (2 DB servers, 2 web servers, firewall and load balancer)
  • Cost: $2850.00 - Monthly, $3700.00 - Setup (average of ~$700/mo per server)
  • Server options 1
    • RHEL ES
    • 1 x 2.4 GHz Xeon
    • 2 x 1 GB ECC
    • 2 x 73 GB SCSI (10k)
    • $1150.00 - Monthly
    • $1000.00 - Setup
  • Server option 2
    • RHEL ES
    • 2 x 3.2 GHz Xeon
    • 4 x 512 MB ECC
    • 2 x 73 GB SCSI (10k)
    • $1580.00 - Monthly
    • $1000.00 - Setup

VeriCenter

  • Michael Lee, Ed Buck, 281-584-4577
  • AppSite Hosting
  • 100% power and network SLA
  • 24/7 service and support
  • Atlanta Data Center is SAS 70 Type II certified
  • Premier 2 package
    • Dell PowerEdge 2850 Dual Xeon
    • 3GB RAM
    • 4 x 73GB Drives R5
  • Redundant Cisco Firewall protection with completely customizable rule sets
  • Red Hat Enterprise Linux operating systems with 24x7 administration and patching services
  • Bandwidth - 1,000 GB of monthly data transfer
  • Email services - 10 domain based addresses per server, 50 MB of storage per box and full anti-virus protection
  • Two hours of monthly support are included in the monthly recurring charge

Netriplex

  • Headquarters in Boston, MA; datacenter in [)http://www.biltmorepark.com Asheville, NC]
  • 25Gb Month Offsite Backup
  • 100% Uptime SLA
  • 24 x 7 Toll Free Technical Support
  • Multi-homed (OC 12) connectivity
  • A Dedicated Account Manager
  • Dedicated Monthly bandwidth transfer for 100GB burstable up to 10Mbps
  • One year term at $450.00/month plus 1 month’s equivalent for setup fee.
  • Server Configuration DELL PowerEdge SC1425:
    • Dual Intel® Xeon™ Processor at 2.8GHz/2MB Cache, 800MHz FSB
    • Red Hat Enterprise Linux
    • 2 Drives Attached to add in SCSI Controller, Software RAID 1
    • Controller Card,SCSI,39320,Internal/External,U3,Low Voltage Differential
    • 2x 73GB 10K RPM Ultra 320 68pin SCSI Hard Drive
    • Hardware RAID possible with upgrade to PowerEdge 1850 for $35/mo
  • Free DNS for up to 5 domains

LayeredTech

  • A high-speed dedicated Internet server
  • Hardware guarantee: Layered Technologies' will replace failed hardware within 2 hours
  • Basic server monitoring: 24 hours a day, 7 days a week, 52 weeks a year*
  • 99.9% SLA
  • Total control of your server with root or administrator access
  • Rack space in Layered Technologies data center
  • Server reboot and/or application shutdown and restart
  • Uninterruptible power management system
  • Uninterrupted air cooling and security system in the data center location
  • Connection to 9 backbone providers
  • 24/7/365 onsite hands and eyes for reboots and assistance
  • FAQ: Our Support Department is here to help assist you in matters that require hands on support or have to deal with issues directly related to the network, power, bandwidth, hardware of the server, reboots and some OS issues. Additional support, or level 2 and above, costs $75/hr.
  • No contracts, 7 day notice
  • Customer is responsible for DNS
  • Sublets space from Savvis datacenter in Fort Worth, TX
  • Dell PowerEdge, $300/mo, $100 setup - servers

Pair Networks

  • FreeBSD and P4 only, no RAID?

ServInt

  • $400/mo, 2x dual core Xeon, 2GB RAM, 2x 250GB SATA RAID hot swappable, 2000GB transfer, overage $1/GB
  • Fedora/CentOS
  • We can also patch and update the Base Operating System, as well as primary applications such as Apache, PHP, and MySQL when necessary, install third-party software, monitor a variety of services on each account, and proactively resolve issues that present themselves. We answer lots of questions, and try to help in every way.

EV1Servers

  • Server: $400/mo, $100 setup, dual Xeon 3.2GHz, 2x73 SCSI, 2500GB transfer
  • RHEL3 only?

Server Central

  • PowerEdge 2950 for $470/mo, $300 setup, 2x Xeon, 4GB RAM, 6x 73GB SCSI in RAID, 1200GB transfer
  • Customers: php.net, arstechnica.com, yousendit.com
  • Based in Chicago, IL
  • Datacenter facilities are provided by Equinix (www.equinix.com) and SARA Computing (www.sara.nl)
  • phone support 24/7
  • redundant network and power

Rackmounted

  • Free 3 hours of support for the 1st 3 months; after that, 1st 30 mins free, $100/hour in 15 min increments (standard/simple support)
  • $450/mo, $100 setup, 2x 3.2 Xeon, 2GB RAM, 1000GB transfer, 2x 160GB ATA
  • Full management:
    • Operating system installation
    • Rebooting
    • DNS Service
    • Free Simple Support
    • Operating system management
    • Software updates
    • Software installations

Peer1 Networks

  • Anthony Taliercio, 866-823-9806, ataliercio at peer1.com
  • RedHat Linux Enterprise - 4.0 ES
  • 2 x Intel Xeon 3.0 GHz, 2GB DDR RAM, 2 x 250 GB SATA Hard Drives - RAID-1, generic hardware
  • $400/mo, no setup
  • Daily Incremental Backup, 30 Day Retention Period: $100/mo, no setup
  • 12-month contract, $500/mo
  • Gold Support (24/7/365 System Admin Level Support, Managed OS Patching)
  • Personal Sales Account Manager
  • 1-Hour Hardware Replacement SLA
  • 100% Network Uptime SLA on PEER 1 Network (over 500 Network Peering Partners, 12 POPs Throughout N America and Europe)
  • Aggregate Data Transfer: 1000 GB per Month, 100 Mbps connection)
  • DNS is included
  • Plesk is an option for $13/mo

Fast Servers

  • Velocity Enterprise DX
  • Dual Xeon 2.8GHz Processors
  • 2GB RAM
  • 2x146GB SCSI HDs
  • RAID 1
  • centOS
  • 1500GB Bandwidth
  • Up to 5 IPs
  • DEFCON 4 Server Management
  • $533/mo, no setup fee
  • basically unmanaged only basic network monitoring
  • Server customization page

Resources

  • Colotraq - bids from managed hosting providers
  • Cheap IP Takeover
  • Distributing Server Load with Round-Robin DNS
  • MySQL Cluster
  • Replication
  • Identifying and Avoiding Dishonest Hosting Providers?
  • Recommend a server hosting company?, December 21, 2005 1:24 PM
    • I like pair.com for stability and generous bandwidth allowance, but I like ev1servers.net because they do not have an adult content policy.
    • try theplanet.com - they're on a wicked fast backbone, and it looks like they offer gamer-specific plans. My experience with them is they're very responsive as well
    • I've been very happy with Layered Tech. Dedicated servers start at around $70/month with 1TB of bandwidth / month. I picked them because they offered Debian and because they offer very little in the way of handholding sysadmin support and are therefore cheaper. If you know how to be root yourself, it's a good option.
    • Second pair.com Ten years, and my server has been down for a total of 23 minutes due to catastophic drive failure. Otherwise, not even a hiccup.
  • Deal discussion
    • theplanet and liquidweb service hit or miss, rackspace - expensive but worth it
  • The host with the most
    • Please do not use 1and1. They're freaking dreadful as many mefites (including myself) have stated many, many times on Ask and the blue.
    • I have to give a vote for EV1Servers. I've tried a lot of companies, but have been with EV1 with several servers for a few years now. Not even a blip in the service during Rita.. amazing reliability.

High availability

URL forwarding

via proxy using mod_rewrite?

DNS failover

  • No-IP Advanced Server Monitoring
    • 24/7 server monitoring, every 5 minutes
    • $150/yr
  • NeuStar (UltraDNS) SiteBacker
    • time to switch: 2 mins to determine if the server is down plus 2-8 mins to propagate the DNS change (3-4 mins on average) - may not be honored by the downstream DNS servers but usually is.
    • 11 DNS clusters around the world, users are directed to the closest one, no single point of failure
    • clients include Amazon, MySpace, Oracle, Sharper Image, etc
    • $375/mo, $500 setup with 1-yr contract ($325/mo, $250 setup if signed by the end of August, 2006)
    • $350/mo, $500 setup with 2-yr contract
    • $300/mo, no setup fee with 3-yr contract
    • 1 domain, 2 resource records, 350,000 queries per month
    • Additional domain $2/mo, resource record: $0.29/mo, 1000 queries: $0.61
  • DNS Made Easy - DNS failover and system monitoring
    • $60/yr
    • 10 million queries per month

IP takeover

works only with servers on local network

Clustering

Wikipedia

  • Wikipedia's solution is to use round-robin DNS to distributed traffic among geographically separated server clusters
    • How does the data (MySQL and images) get synchronized between clusters?
    • Do web servers read data from MySQL slaves and write data to the master?
    • Is SAN mirroring used?

Performance tuning

  • Slashdotting - estimates put the peak of the mass influx of page requests at anywhere from several hundred to several thousand hits per minute.
  • From Slashdot FAQ on 7/12/04
    • slashdot typically serves 80 million pages per month. We serve around 3 million pages on weekdays, and slightly less on weekends.
    • thousands of hits at the site in minutes
  • OWW serves ~40,000 pages/day (1.2 million pages/mo) as of August 2006
    • ~75x difference with Slashdot
    • ~25% of OWW traffic is created by spiders/bots (mostly google)
    • server load at peak times is around 25-30%, at normal times: 5-10% (as reported by sar)
    • caching would give 2-10x boost, LAMP tweaks can give some boost as well
    • server can handle ~4x traffic in the current software/hardware configuration
    • therefore we have 8-40x performance reserve on current hardware simply by tuning the software but this needs to be confirm by benchmarking
  • static version of the site would be able to handle slashdot-style traffic surges easily
  • ~25 changes/hour (50 changes total) between 4 and 6pm on 2006-08-18 (Friday)
  • Surviving a slashdotting with a celeron 466
    • I upped the number of min and max servers, set a server timeout of 15 seconds, and made sure new server processes were started every 500 requests or so to keep runaway processes from killing the server.
    • My first course of action was to figure-out how to make the page static so that it wouldn’t have to hit MySQL each time someone requested that page. (dewikify whole site?)
    • One last thing I did was kill all unnecessary processes such as SMTP
    • Once I put the box back in, it took a bit more Apache tweaking to get it perfect, but eventually my spare parts Celeron 466 was handling a Slashdotting at only 30% CPU utilization!
    • Usually I do around 2-2.5 GB of web traffic per month, but in one day I did over 4 GB!
  • Configuring Apache: Don't Succumb to the Slashdot Effect
    • original article
    • list of Apache settings and their suggested values
    • MaxClients 500 -- that one you have to be careful with. You need to do tests where you actually consume the max clients to see if you start to swap and/or hit an io wait bottleneck before you get there, which you almost certainly will with many types of dynamic pages running with 500 clients. For example, if you have php pages averaging 8Mb allocation, 500 clients means you're using 4Gb just for php.

Separate server for static and dynamic content using mod_proxy and mod_rewrite http://www.onlamp.com/pub/a/onlamp/2004/02/05/lamp_tuning.html http://phplens.com/lens/php-book/optimizing-debugging-php.php

  • As your Site Grows - article on Wikimedia
    • 1 x 4GB Squid
    • 5 x 512MB 3GHz P4 Apaches
    • 1 x 4GB dual Opteron database server with 6 fast disks in RAID 10.
    • One or two of the Apaches will have extra RAM - up to two or three gigabytes per computer

One set like this should handle about 150 to 200 page views per second. For press links and slashdotting the Squid should handle at least twice that (300 to 400 views/sec?).

  • Problems
    • Apache spawning too many children (out of memory)
  • Solutions
    • Caching helps
      • PHP cache (precompiled objects)
      • statis page cache (files)
    • Make all/some pages static (reduce PHP/wikitext parsing and DB access)
    • Slashdot effect: Assistance and prevention - Wikipedia article: Coral, redirect to mirror nets

Hardware

  • Opteron 246 runs at 2GHz and is roughly equivalent to Intel Xeon 3.2GHz
  • Dell PowerEdge 2850 review by PC Magazine
  • Hardware (vs software) firewalls
    • reduce load on the server
    • provide VPN capabilities
    • keep probes away from the servers

Rackspace alternatives

  • Peer1 (yes, they do managed servers), Colo4Dallas and Carpathia Hosting
  • Softlayer
    • $400/mo
    • unmanaged - won't restore files in the event of drive failure?
  • Hostway (same as NetNation?)
    • $400 for Dell PowerEdge 2850, $350 setup
  • LiquidWeb
    • good comments generally but 30 min outage due to lightning described here
    • 100% Network Uptime Guarantee
    • 2 Hour Hardware Replacement SLA
    • 2400GB Premium Multi-Homed Bandwidth (1200in and 1200out)
    • $400 dual Xeon or dual core dual opteron + $120 for scsi raid, no setup fee
    • CentOS
    • 50GB remote backup free
Retrieved from "https://openwetware.org/mediawiki/index.php?title=OpenWetWare:Infrastructure&oldid=65989"