The nature of OSR ability scores, and the shift away from an even bonus progression, makes relying solely on modifiers problematic. Plus, random rolls are fun.
This is one possible compromise. It's a little bit fiddly, but it's both mostly fair and mostly random, at least as much as I can make it. It also offers another way of approaching demi-human ability bonuses. I don't know if I'll actually use it, but I thought I'd offer it up as a thought exercise and a slightly different spin on things.
Initial Ability Scores
Generating ability scores is a two-step process. Each character has 3 points in ability bonuses, which can be distributed as the player desires. No starting bonus can exceed +2 (so either +2/+1, or +1/+1/+1). At the GM's option, characters can assign a -1 penalty to one score in exchange for either an additional point or increasing the cap on one score to +3.Bonuses and scores line up as follows:
Example: Shannon wants to play a knight. She puts a +2 into Strength, and a +1 into Constitution. Everything else is +0. Her ability scores are not yet determined, but Strength is 16-17, Constitution is 13-15, and the other scores are in the 9-12 range.
Next, the player rolls a d6 for each ability, and consults the following chart for the exact ability score.
Example: Shannon rolls 2,4,1,2,2,5. The "2" roll in Strength gives her character a final score of 16, the "4" gives her a 14 in Constitution, the "1" gives her Dexterity of 9, the "2" gives her a 10 in Intelligence and Wisdom, and the "5" gives her a Charisma of 11. Final tally: Str 16, Con 14, Dex 9, Int 10, Wis 10, Chr 11.
Demihumans can be handled in one of two ways. They receive the same number of bonus points as humans, but can either have a higher cap in one score (+3 instead of +2) and a lower cap in another (+1 instead of +2) OR they can have a +2 bonus on one d6 roll, and a -2 penalty on another. The GM should decide which method, and which scores are affected.
Example: Shannon decides to play a half-orc. Half-orcs get a +2 to determine their Strength, and a -1 to determine their Charisma and Intelligence. Using the numbers above, this is sufficient to increase her Strength to 17 (4+2=6), drop her Intelligence to 9 (2-1=1), and doesn't change her Charisma (5-1=4, which is still good enough for an 11).
Advancing Ability Scores
Advancing ability scores works exactly as step two, above. The player rolls a d6 and compares the result to her existing score. If the d6 result would garner a better score, the score advances by 1. If not, the score is unchanged. If a score is maxed out in its range (ie, a score of 12, 15, or 17), a result of 6 or higher bumps the ability to the next range (maxing out at 18). The frequency and quantity of these attempts is up to the GM, and could range from not at all, to one ability every other level, to all abilities every level.If the second demihuman option is used, the modifiers are applied as appropriate. This means that poor ability scores (those with penalties) will never advance to the next tier (since they can never have a "6")
Example: Shannon gets to roll for all of her abilities, and rolls 1,5,1,5,2,5. The "1" obviously doesn't advance her Strength, whether she's a half-orc or not. Her Constitution, Dexterity, and Wisdom likewise remain the same. The "5" result in for Intelligence is sufficient to improve her score to a 10, whether she's a half-orc or not (with a score of 9, any result of 2 or higher would have bumped her Intelligence up), but because her Charisma score is already 11, a "5" is not sufficient to increase it to a 12. In the final reckoning, nothing has changed except her Intelligence, which increased from 9 to 10. That might be a slight improvement if she was a magic-user, but not to a knight. All of her modifiers remain the same.