1

I have checked that the data are stored on text file in cosmos as I show here:

hadoop fs -cat /user/cristina.albaladejo/def_serv/def_servpath/sensorreading_1_sensorreading/sensorreading_1_sensorreading.txt

{"recvTime":"2015-05-06T13:33:20.905","timestamp":"2015-04-15 11:10:00", "timestamp_md":[],"value":"-50.6", "value_md":[],"sensortypeid":"1", "sensortypeid_md":[],"nodeid":"1", "nodeid_md":[],"systemid":"1", "systemid_md":[]}
{"recvTime":"2015-05-06T13:34:00.784","timestamp":"2015-04-15 11:10:00", "timestamp_md":[],"value":"-220.2", "value_md":[],"sensortypeid":"1", "sensortypeid_md":[],"nodeid":"1", "nodeid_md":[],"systemid":"1", "systemid_md":[]}

But when I enter in hive, there is not any table with my name. Why not the hive tables are created? My file of configuration of Cygnus is the following:

cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink
cygnusagent.channels = hdfs-channel mysql-channel

cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 10
cygnusagent.sources.http-source.interceptors = ts de
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
cygnusagent.sinks.hdfs-sink.cosmos_default_username = cristina.albaladejo
cygnusagent.sinks.hdfs-sink.cosmos_default_password = my_passw
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
cygnusagent.sinks.hdfs-sink.attr_persistence = column
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
cygnusagent.sinks.hdfs-sink.hive_port = 10000
cygnusagent.sinks.hdfs-sink.krb5_auth = false
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf

# OrionMySQLSink configuration
cygnusagent.sinks.mysql-sink.channel = mysql-channel
cygnusagent.sinks.mysql-sink.type =     es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
cygnusagent.sinks.mysql-sink.mysql_host = 130.206.80.46
cygnusagent.sinks.mysql-sink.mysql_port = 3306
cygnusagent.sinks.mysql-sink.mysql_username = root
cygnusagent.sinks.mysql-sink.mysql_password = my_passw
cygnusagent.sinks.mysql-sink.attr_persistence = column

# hdfs-channel configuration
cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
frb
  • 3,738
  • 2
  • 21
  • 51
Cristina
  • 161
  • 1
  • 1
  • 9
  • 1
    https://github.com/telefonicaid/fiware-cygnus/issues/323 – Ajay Pandya May 08 '15 at 11:05
  • 1
    http://stackoverflow.com/questions/27918545/automatic-creation-of-the-hive-tables-on-cosmos – Ajay Pandya May 08 '15 at 11:06
  • As Ajay is pointing, this is most probably due to the well-known libthrift library issue. How did you install Cygnus? Using the RPM or from sources? – frb May 11 '15 at 14:13
  • I have copied the file libthrift-0.9.1.jar in /usr/local/apache-flume-1.4.0-bin/lib but hive tables are not created. I installed Cygnus wit the command yum install cygnus – Cristina May 12 '15 at 07:26
  • I copied the libthrift-0.9.1.jar library in the directory /usr/local/apache-flume-1.4.0-bin/lib, I restarted the machine, I rebooted Cygnus and modified the attribute value (in this order). The data is stored in the .txt but hive table is not created... Do you know what is happening? – Cristina May 13 '15 at 08:24

0 Answers0