[Deploying Sakai] Deployment sizing question

[Adams, David]

Just thought I'd mention Virginia Tech's setup. We're smaller than
Florida or Michigan--about 30,000 students. We're not full-scale on
Sakai yet, but the transition from Blackboard is underway. This Spring,
at peak times, we've been hitting intermittent peaks of around 1000
unique sessions per hour. (Compare this to peaks of around 8000 or more
for Blackboard.)

Adi wrote:
> we've found that appservers are not cpu bound, and now
> with 64-bit machines and 6GB heaps are not memory bound
> -- for our load, we have 5 appservers and they are overkill,
> though they are necessary to spread the risk of Sakai going
> unresponsive (for many different known and yet-to-be-determined
> reasons)

This all fits our experience so far on Sakai, but also with Blackboard.
We've only been running one production Sakai app server this academic
year (an early-2007 dual-quad-core Dell 1950 with 8GB), and it has not
been stretched CPU- or memory-wise. We'll be adding a second machine
when we do an upgrade this weekend, but as Adi says, at this point it's
only for reliability. The only performance problems we've had we can
trace back to the database (more on that below).

Todd wrote:
> > What network load balancers are being used to distribute load across
> > their application clusters?

We use an F5 BIG-IP LTM 3400, and we're upgrading to a failover-pair of
LTM 1600s this summer. Even with the load balancer handling SSL, our
Sakai + Blackboard load is nowhere near its capacity.

Todd wrote:
> What methods are being used for node affinity within the load
> balancer? ie. stickytimes, cbr/cookie persistence, something else?

We do cookie injection, which is basically a turnkey setting on the load
balancer.

Todd wrote:
> So do you have webdav as a window into your binary file storage?  
> conjunction with that question, do you keep binary files in your 
> database or do you have a separate file server for binary objects?
> [...] I'm interested if you folks have a strong preference 
> one way or another on this issue?

Sakai provides a WebDAV server that maps into its content system. The
actual file structure on disk is not designed to be interacted with. In
fact, I think it's at least partially designed to discourage attempts at
direct manipulation. All metadata and hierarchy information are stored
in the database. There are a few large installations that keep the file
objects in their Oracle, but I would recommend not going that route
unless your Oracle folks are able to maintain multi-terabyte schemas
easily. To me, it seems that for both performance and practical backup
reasons, using the filesystem is a no-brainer, and honestly, I believe
the main reason DB storage is available at all is for easy demoing.

Like many folks, we store our filesystem content on an NFS share hosted
by the university's enterprise NAS (a NetApp). Our Oracle instance uses
raw SAN space on another storage system.

Todd wrote:
> What are the specs on your production DB server?

Right now we're running Oracle 10g under RedHat on a three-year-old Dell
6850 with four 3.6GHz single-core Xeons and 20GB of RAM. However, we've
run into two or three situations in which some particular operation in
Sakai manages to overpower it. There's a lot of tweaking we can probably
learn how to do, but since we're ramping up, we have purchased new
hardware for an upgrade this summer: an IBM x3755 with 64GB and 4x
quad-core 2.4GHz Opterons.

We used to run Blackboard's Oracle on Sun hardware, but we did some
performance comparisons two years ago and we found that a $16k Dell 6950
could handle twice the load of a Sun costing five times as much, so
we've happily switched to AMD hardware.

David Adams
Director of Learning Systems Integration and Support
Virginia Tech Learning Technologies