Using the Mechanical Turk to validate petition signatures

Jeff sez:

To qualify our initiative for the ballot in Seattle, we need the signatures of more than 20,600 registered city voters. This means we're going to have approximately 2,000 - 3,000 pages of handwritten petition forms.

We wrote two Mechanical Turk tasks to digitally capture the names, addresses and emails of petition signers and then to have workers verify what percentage of the signers are actually registered to vote in Seattle (as only these voters count towards our ballot qualification requirements).

Paying anonymous Internet-based Mechanical Turk workers does raise some murky questions related to outsourcing vs. local employment. And, since our Initiative is aimed at re-establishing democracy and social justice, we are sensitive to this. Thus far, we've just experimented with the Turk system. We're not sure yet if we'll use it in whole. And, if we do, we're going to be very thoughtful about how much we pay for each task.

Jeff's project is "initiative 103":

Initiative 103 will change the law in Seattle to:
* Ban corporate spending on elections, reversing Citizens United
* Ban corporate lobbying except in public forums
* Strip Corporate Personhood and judge-made corporate "Constitutional" rights

Using Mechanical Turk to Digitize Handwritten Petition Forms and Validate Voter Registrations

30

  1. it raises more than just questions about local employment… what about privacy for the people signing the petition? often those petitions include address info to verify your eligibility.  can you ensure data privacy when hiring somebody anonymously on the internet to verify identity?

    knowing this would make me wary of signing any petitions.

  2. it raises more than just questions about local employment… what about privacy for the people signing the petition? often those petitions include address info to verify your eligibility.  can you ensure data privacy when hiring somebody anonymously on the internet to verify identity?

    knowing this would make me wary of signing any petitions.

    1. Who cares?  Your name and address are already freely available in the phone directory etc.  The reflexive anxiety about identity theft and privacy is often overblown.

        1.  yes…  so quickly and efficiently discover every petition that i’ve signed since 1993 and cross correlate that with my library check-out record and credit-card info.  you can’t, “quickly and efficiently”, can you?  and i put it to you that it is still a difficult matter if you’re “one of them” (nebulous designation, i know, but we each have our own “them”).   do you want to make it easier for your “them”?   even Kafka’s castle dwellers bemoaned the amount of paperwork.

        2.  yes…  so quickly and efficiently discover every petition that i’ve signed since 1993 and cross correlate that with my library check-out record and credit-card info.  you can’t, “quickly and efficiently”, can you?  and i put it to you that it is still a difficult matter if you’re “one of them” (nebulous designation, i know, but we each have our own “them”).   do you want to make it easier for your “them”?   even Kafka’s castle dwellers bemoaned the amount of paperwork.

          1. And having some random person corroborate your name against a record of your address will change this how?

            come to think of it, what do you think the name is being matched to?  Probably some sort of existing record of your address, huh?

    2. The petitions are almost always a matter of public record. I guess you can feel better knowing that typically the only ones who will exercise their right to see the petitions are operatives of the political parties rather than mechanical turk users whose eyes are probably wanting to pop out of their head from the mind numbingly boring task they’re subjected to.

      1. Similarly, voter registration info is publicly available.  Any “candidate” can get a list of e-mail addresses.

  3. Why bother with this?  The usual approach signature gatherers take is to collect substantially more signatures than they need, so they still have enough after invalid signatures are disqualified. How many extra do you typically need?  Just ask organizations who’ve done signature-gathering in your area before.  It’s less expensive and less privacy-intrusive to just get more signatures, and Mechanical Turk is piece labor that pays people sub-minimum wage anyway.

    1.  Many petitions pay the gatherers by the number of signers they get.  Lately there’s been a more than few gatherers for whom a substantial number of the pages after the first few are largely fictitious.   To the degree that there have been recent cases where a substantial proportion of all the signatures are made-up and/or copies-of-other-petition drives.  This is the problem this sort of effort may address.  That is, one might have to collect a lot more than “more than you need”.  …against, ‘diminishing returns’, i think.

  4. Aren’t all the signatures going to need to be re-validated after they’re turned in? I don’t imagine the Mechanical Turk system will allow anyone to notarize the results.

    So if they’re going to be re-validated anyway, why not do as Joe Buck says and just get n% more signatures?

    Possible reason off the top of my head: n% varies wildly across different ballot initiatives, and the extra cost to gather 30% more signatures instead of 10% more signatures is such that it’s worth spending money to check that maybe you’re fine after just 10%.

    1. There may also be a significant turn around time for the validation process after the petitions are turned in.  If you usually go 8% over for safety but this time you get 11% bad entries, you need to know as soon as possible so you can get the additional required signatures before the deadline.

    2. There may also be a significant turn around time for the validation process after the petitions are turned in.  If you usually go 8% over for safety but this time you get 11% bad entries, you need to know as soon as possible so you can get the additional required signatures before the deadline.

  5. In response to the above comments, all the petitions will be public record in a few months when we turn them in. We’re NOT paying any signature gatherers – this is an all volunteer effort. So, it’s not easy to just gather more signatures. One alternative we’ll consider is asking volunteers to digitize petitions but Turk is a more automated/on demand option.

  6. Paying anonymous Internet-based Mechanical Turk workers does raise some murky questions related to outsourcing vs. local employment

    God, I am sick of this kind of mealy-mouthed language. It doesn’t “raise murky questions,” it represents a very clear and obvious moral and ethical failure in direct contradiction to your ideals.

    1.  Exactly.  Using Mechanical Turk says you’re happy to exploit the unemployed and the poor to do piece labor for you for far less than minimum wage.  This is not a progressive thing to do.

      1.  As an alternative, you could ask for financial contributions that would allow you to hire someone to check signatures and pay that someone a living wage.  But increasingly, even progressive nonprofits pay the staff that does the work little to nothing but pay the director six figures.

      2.  As an alternative, you could ask for financial contributions that would allow you to hire someone to check signatures and pay that someone a living wage.  But increasingly, even progressive nonprofits pay the staff that does the work little to nothing but pay the director six figures.

    2. “We’re a progressive organization, so we understand the ethical issues here. But we’re going to go ahead and pay people in India $3 an hour to work for us anyway. Please sign our petition!”

  7. Sedan, Joe, well said. As I’ve said, we did this as a proof concept thus far and haven’t chosen to implement it broadly. 

    Some of the steps here may be applied to volunteers as well … e.g. we can host the signatures, use google doc spreadsheets and have specific volunteers do online tasks.

    If you’d like to contribute funds for us to hire locally, you can do this by clicking the donation button on the right sidebar of http://i103.org.

    1. Why would outsourcing jobs to skirt minimum wage laws even be a concept in the first place? You’re aware that the tech novelty of the approach doesn’t affect the ethics of the approach, right?

        1. Just that you need specific contributions to pay people locally. If you’re going to pay a reasonable wage, then why do you need Turks at all? The only difference is losing direct control over the process.

          1. The automation and on demand nature of Turks is either than training and deploying volunteers … and in the absence of funding for local workers (or office space)… MTurk is a compelling option. It’s also ideal for repetitive tasks that you wouldn’t want one person to be forced to do 20,000 times … but are fine to have 20,000 people do once.

          2. It’s also ideal for repetitive tasks that you wouldn’t want one person to be forced to do 20,000 times … but are fine to have 20,000 people do once.

            I wish you had just said that in the first place. I would highly suggest you do your research. You’re going to find out pretty quickly that that’s not how crowdsourced-labor works, in reality. You’re not going to have masses of bored kids and the unemployed chipping away at your problem, you’re going to have an outsourcing/insourcing firm in India or China paying employees poverty wages to do it for you.

  8. Maybe some civics classes before worrying abut the workers. Will a petition, initiative, what have you, passed or not, have any power over a supreme court decision? No.

  9. Tim, take a look at http://envisionseattle.org/2012/04/pacific-northwest-emerging-as-a-stronghold-for-a-new-kind-of-activism.html and http://envisionseattle.org/2012/05/supreme-court-approvals-drop-to-25-year-low-65-percent-disapprove-of-citizens-united.html

Comments are closed.