Archive for November, 2019

One month with the reMarkable tablet

A month has passed. Time for an update on my experience with the reMarkable tablet. Here’s what’s happened since my last blog post on its use, plus a few comments about workflow, feature use, and improvements that I hope the company will consider.

I replaced the first nib. For several days my writing experience was strange, the tablet seemingly more fidgety and sensitive. I finally had another look at the new nib and found it had not been inserted it all the way. Problem solved.

I rsync the tablet every few days to my desktop. Eventually that will be a daily process, with weekly new full backups and daily incremental updates. Cleanup work on the forked script for conversion of notebooks to pdfs is ongoing.

reMarkable tablet with custom sleep screen

I leave WiFi off except during rsyncs; I haven’t yet attempted to convert any pdfs to text. I’d be using an out of band means to do that in any case, since I don’t want to save data on the company’s servers.

The problem with pages in the thumbnail view being cropped on top has still not been resolved. I make allowances by writing a title lower down in the page. It’s not great. (Where’s that open issue tracker, folks?)

I’m getting into the routine of things with my daily todo lists. At some point I’ll probably have notebook blanks with dates already filled in, in big bold text that appears clearly on the thumbnails, and with standard todo items above the date, to be duplicated for each half of every month.

I’ve not yet played with using layers to display titles in large text and free up the virtual lined paper for notes.

I have a thingiverse 3-d printed stylus clip (see https://www.thingiverse.com/thing:2950560) which fits well and prevents the pen from rolling around on the smooth table surface. No folio yet, as I’ve not needed to travel with the tablet and have not seen a solution I like. I’d prefer a book-style cover that is both slim and does not require the tablet to be glued into the cover. Anybody have suggestions? DIY acceptable!

I take scratch notes on various topics while working through items in my todo lists. These eventually get cleaned up and exported into separate notebooks or added to existing ones. Moving out multiple pages is a bit tedious; that’s another feature that would be nice to have.

At some point it will be ok to move non-agenda notebooks off to the laptop; these should be converted to pdfs. That’s more incentive to get the conversion script working properly. I need to investigate the various pdf conversion tools and see what produces clean output as well as

pdfs of a reasonable size.

If I could add the pen switcher menu to the pen/pencil/marker in the small menu display, as well as the thumb view on top, I could stay in small or no menu view almost all of the time, instead of switching to large menu view to get these items and then closing that again. Consider it a step closer to the ‘no clutter’ intended design of the device.

A couple of times I have touched something in passing while holding the tablet with a notebook page displayed and the full menu open, that wiped the page. A couple of undos restored it, but it would be nice to know what cleared the page in the first place.

Sometimes the previous page and next page buttons at the bottom of the tablet don’t respond on first try and I need to press them again. Perhaps this is a sensitivity issue. It’s not a huge drawback, more of a small annoyance, but something that should be ironed out for future shipments of the device.

At this point, I don’t have any desire for other apps on the device except for rsync and some way to manage custom templates, splash screens and so on. Sure, everyone can roll their own but that’s a waste of hours of fiddling around by a lot of users. These items should be easy enough for the company to fold into the firmware.

I’m a little bit anxious about what will happen when the battery starts to fail to hold a charge in a couple of years, because it’s not intended to be replaceable. I’m not one of those folks that gets a new phone every couple of years; devices should be built to last and to be repaired and used as long as possible, rather than regularly added to the landfill. Maybe by then, a third party battery will be available and we’ll be able to manage replacements ourselves, even if completely unauthorized.

I have not once used the tablet to read a pdf or epub; that happens on the laptop. I don’t have commute time that gets filled by reading, and coffee shops are for socializing. We’ll see if that’s changed by the 6 month mark.

I still haven’t cracked open the paper notebooks even once. Maybe when I go somewhere without power for longer than a few days 🙂

Digging into Structured Data for Media on Commons (Part 2)

Now that we know all about what MediaInfo content looks like and how to request it from the api, let’s see how to add MediaInfo content to an image. If you don’t remember all that, have a look at the previous blog post for a refresher.

Adding captions

Captions are called ‘labels’ in Wikibase lingo, and so we’ll want to put together a string that represents the json text defining one or more labels. Each label can have a string defined for one or more languages. This then gets passed to the MediaWiki api to do the heavy lifting.

Here’s an example of the ‘data’ parameter to the api before url encoding:

data={"labels":{"en":{"language":"en","value":"Category:Mak Grgić"},
"sl":{"language":"sl","value":"Mak Grgic"}}}

You’ll need to pass a csrf token which you can get after logging in to MediaWiki, and the standard parameters to wbeditentity, namely:

action=wbeditentity
id=<Mxxx, the MediaInfo id associated with your image>
summary=<comment summarizing the edit>
token=<csrf token you got from logging in>

Since I spend most of my coding life in Python land, we love the requests module. Here’s the relevant code for that:

        params = {'action': 'wbeditentity',
                  'format': 'json',
                  'id': minfo_id,
                  'data': '{"labels":{"en":{"language":"en","value":"' + caption + '"}}}',
                  'summary': comment,
                  'token': self.args['creds']['commons']['token']}
        response = requests.post(self.args['wiki_api_url'], data=params,
                                 cookies=self.args['creds']['commons']['cookies'],
                                 headers={'User-Agent': self.args['agent']})

where variables like minfo_id, comment and so on should be self-explanatory.

You’ll get json back and if the request fails within MediaWiki, there will be an entry named ‘error’ in the response with some string describing the error.

You can have a look at the add_captions() method in https://github.com/apergos/misc-wmf-crap/blob/master/glyph-image-generator/generate_glyph_pngs.py for any missing details.

Since that’s all pretty straightfoward, let’s move on to…

Adding Depicts Statements

A ‘depicts’ statement is a Wikibase statement (or ‘claim’) that the image associated with the specified MediaInfo id depicts a certain subject. We specify this by using the Wikidata property id associated with ‘depicts’. For www.wikidata.org that is https://www.wikidata.org/wiki/Property:P180 and for the test version of Wikidata I work with at https://wikidata.beta.wmflabs.org it is https://wikidata.beta.wmflabs.org/wiki/Property:P245962 so you’ll need to tailor your script to the Wikibase source you’re using.

When we set a depicts statement via the api, existing statements are not touched, so it’s good to check that we don’t already have a depicts statement that refers to our subject. We can retrieve the existing MediaInfo content (see the previous blog post for instructions) and check that there is no such depicts statement in the content before continuing.

When we add a depicts or other statement, existing labels aren’t disturbed, so you can batch caption some images and then go on to batch add depicts statements without any worries.

The MediaInfo depicts statement, like any other Wikibase claim, has a ‘mainsnak‘, and a ‘snaktype‘ (see previous blog post for more info). Crucially, the value for the depicts property must be an existing item in the Wikidata repository used by your image repo; it cannot be a text string but must be an item id (Qnnn).

Here is an example of the ‘data’ parameter to the api before url encoding:

data={"labels":{"en":{"language":"en","value":"Template:Zoom"},
"fa":{"language":"fa","value":"الگو:Zoom"}},
"sitelinks":{"commonswiki":{"site":"commonswiki",
"title":"Template:Zoom"},"fawiki":{"site":"fawiki",
"title":"الگو:Zoom"}}}

For the requests module, you’ll have something like this:

        depicts = ('{"claims":[{"mainsnak":{"snaktype":"value","property":"' +
                   self.args['depicts'] +
                   '","datavalue":{"value":{"entity-type":"item","id":"' +
                   depicts_id + '"},' +
                   '"type":"wikibase-entityid"}},"type":"statement","rank":"normal"}]}')
        comment = 'add depicts statement'
        params = {'action': 'wbeditentity',
                  'format': 'json',
                  'id': minfo_id,
                  'data': depicts,
                  'summary': comment,
                  'token': self.args['creds']['commons']['token']}
        response = requests.post(self.args['wiki_api_url'], data=params,
                                 cookies=self.args['creds']['commons']['cookies'],
                                 headers={'User-Agent': self.args['agent']})

Note here that while we retrieve MediaInfo content from the api and these entries are called ‘statements’ in the output, when we submit them, they are called ‘claims’. Other than that, make sure that you have the right property id for ‘depicts’ and you should be good to go.

There are some details like the ‘<code>rank:normal</code>’ bit that you can learn about here https://commons.wikimedia.org/wiki/Commons:Depicts#Prominence  (TL;DR: if you use ‘rank:normal’ for now you won’t hurt anything.)

Again, the variables ought to be pretty self-explanatory. For more details you can look at the add_depicts method in the generate_glyph_pngs.py script.

You’ll get an ‘error’ item in the json response, if there’s a problem.

More about the sample script

The script is a quick tool I used to populate our testbed with a bunch of structured data; it’s not meant for production use! It doesn’t check for nor respect maxlag (the databases replication lag status), it doesn’t handle dropped connections, it does no retries, it doesn’t use clientlogin for non-bot scripts, etc. But it does illustrate how to add retrieve MediaInfo content, add captions, add items to Wikidata, and set depicts statements.

Much more is possible; this is just the beginning. Join the #wikimedia-commons-sd IRC channel on freenode.net for more!