It's a sporting analogy. For example, football teams belong to various "leagues" or grouping based on how many games they win. The speaker is suggesting that they do not "belong" to the supposedly superior group or class of people; they feel "out of their league".

In U.S. baseball, the professional teams people follow are the "major leagues" or the "big leagues," such as the Boston Red Sox. There are also minor leagues--for example, the same club runs the Pawtucket Red Sox. And then there are amateur leagues, and so forth. "I'm out of my league" suggests someone is in fact a baseball player, but he's only a minor league player and he's playing in a major league game and thus isn't qualified to play at that level. Figuratively, it could mean "I do know a little about this topic, but you are asking questions that only a real expert could answer;" or "I do this kind of job, but you're asking me to do a much more complex or difficult job than I know how to do." It could also mean that a person works in a different area of a specialty. For example, you might ask a friend "You're a lawyer, tell me if I can copyright this." She might say "That's out of my league; that's a question about copyright law, and I'm an estate lawyer." Other idioms with similar meanings: "I'm out of my depth here," or "That's above my pay grade."