Media mirrors, om nom nom

People who are close followers of the xml data dumps mailing list will already know that we have a mirror hosting all current wikimedia images at your.org, and that we host bundles of media per project generated every month as well.  (See the mirrors list at Meta for the links.)

Right now the mirror and the downloadable media bundles are hosted off-site; in fact the bundles are generated off-site!  But that’s due to change soon.  We have a couple of nice beefy servers that are going to be installed this week just for that work.

Because the media bundles contain all media for a given project, and many files are re-used across multiple projects, there is a lot of duplication and a lot of wasted space.  Got a couple ideas in mind for cleaning that up.

The other exciting thing happening in media-land is the move to distributed storage (Swift) for originals of all media we host.  Once that happens we’ll need to be able to keep our mirror in sync with the Swift copy.  I’m hoping for testing to be entertaining instead of really really annoying. 🙂  We shall see…

Advertisements

0 Responses to “Media mirrors, om nom nom”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





%d bloggers like this: