I would begin by looking at other protocols in existence, and seeing where there are gaps in functionality and performance, as well as things that are 'must haves'.
Next I would look at the protocol specifications (where available) for ones that are close to what you want to do. This will help you judge scope, complexity, etc. for your own design.
At this point, you should start defining your own requirements. Are you focussing on home environments, or industrial environments? Do you want it to be fast or robust? Do you want to use existing hardware and bitbang a new protocol over that, or design completely new hardware? How much functionality resides in hardware, and how much in software?
Following requirements, would be specifications. This would include things like speeds, distances, RFI specs, error correcting, physical design parameters, etc.
At this stage, you should be at a point where you can start playing with proof of concept designs (hardware and software), and adjust your specifications based on results achieved.
As for actually convincing other people to use your design, unless it's absolutely mindblowing (and even then there are no guarantees - see VHS vs Betamax), you may have to seek professional advice from people who have done similar things.
I would also involve the embedded community when you start getting an idea of what you're aiming for. That should give you some idea of whether you're reinventing some existing wheel, or whether you may actually be filling a niche that nobody has had the guts to tackle.
Having said all this
Developing a robust hardware layer is an extremely time and resource intensive process, considering there are numerous good alternatives available. Maybe focus on building a useful software layer first, which you can always later adapt to a different (yours?) hardware layer.