To answer the second part of your question s first: you can get the current application with "the frontmost application", for example
tell application "Finder"
set frontApp to the path to the frontmost application
end tell
The short answer to the first part of your question is: it is currently impossible to do what you want. Long answer follows below.
From your comments, I understand you are using Python with TCL/TK to build the GUI.
Not every application is able to set the "contentsChanged" indicator (apparently, this is what it is called in RealBasic but not in Apple's API). Check the AppleScript dictionary of the application. Relying on what I read elsewhere, TextEdit used to be able to set the "modified" property of its documents but when I try this in Mac OS X 10.6.8 it no longer works.
If you're using a native Cocoa application, you might be able to set this indicator by adding and removing a space to the document with some GUI scripting, e.g. type a space and use the Undo menu item to remove it. Unfortunately, when I try this in TextEdit, the Undo command also removed the indicator.
I checked Apple's API documentation and there appears to be a method SetDocumentEdited in the NSWindow class. Unfortunately, Apple doesn't describe properly what it does, but it appears to set the dark spot in the red close button if the DocumentEdited property is set to true. You can read about it here. If you're using TCL/TK, it would have to be able to call the SetDocumentEdited method somehow.
You can't tell the operating system to change the DocumentEdited property of a window (which would be the equivalent of the "modified" property you found in an AppleScript dictionary). The (Python-TCL/TK) application you're talking to needs to have this implemented. If TCL/TK doesn't have a command for this, then it is probably impossible, no matter if you use AppleScript or Python or something else. I'd suggest you ask on a TCL/TK forum if someone can write and compile a kind of plug-in to implement this feature.