Titan数据库快速入门之决战云巅

Titan数据库快速入门之决战云巅
强烈推介IDEA2020.2破解激活,IntelliJ IDEA 注册码,2020.2 IDEA 激活码

from:https://rednum.cn/ViewListAction?method=detail&dataid=317&classfyid=1

前言

因公司业务发展,需要建立完整的关系图谱检索系统。经各种比较与考量,我们采用Titan图形数据库来处理复杂的逻辑关系。上一篇《Titan数据库快速入门之神的光芒》同事已经详细介绍了Titan数据库的安装与使用,这里不再赘述。此篇博客介绍Titan的实际应用与成果展示,方便读者更好的理解Titan。

项目背景

本项目基于工程大数据进行研究,下图为Hbase中的部分数据展示。通常,一个工程项目有一个或者多个标段,这些标段分别由不同的公司完成,也存在同一标段由多家公司联合完成的情况。例如,图中所示,上海公路桥梁(集团)有限公司完成了合肥至六安高速公路这一项目的路基十三标的施工。而合肥至六安高速公路下面的剩余标段分别由其他家公司完成,因此,施工公司、项目、标段就串成了一个关系图谱,用Titan数据库存储时对应图中的边和点关系。

Titan数据库快速入门之决战云巅

项目实施

1、关系图谱存入Titan数据库中

Hbase中已经存入了近几年多家工程公司的业绩信息,但每个施工业绩都是独立的,没有建成一张关系图谱。所以首先,根据采集的数据构建一个关系图谱存入Titan数据库中。虽然是Titan数据库,但我们采用Hbase进行存储。
下面代码完成的任务是:
从表CompanyInfos_test中遍历所有公司的业绩信息,获取该公司名字,存为结点v1,并添加属性group=1;获取标段,存为结点v2,group=2;并在公司和标段之间建立一条边;获取项目名字,存为结点v3,group=3;并在项目和标段之间建立一条边;重复上述过程,即可建立一张所有公司、标段、项目串起来的关系图表,并存入Hbase中。

Java代码 

 
 
  1. /*
  2. * To change this license header, choose License Headers in Project Properties.
  3. * To change this template file, choose Tools | Templates
  4. * and open the template in the editor.
  5. */
  6. /*
  7. * To change this license header, choose License Headers in Project Properties.
  8. * To change this template file, choose Tools | Templates
  9. * and open the template in the editor.
  10. */
  11. package com.rednum.graph;
  12. import com.google.gson.Gson;
  13. import com.google.gson.internal.LinkedTreeMap;
  14. import static com.rednum.graph.CompanyCountry.conf;
  15. import static com.rednum.graph.TiTanDB.INDEX_NAME;
  16. import static com.rednum.graph.TiTanDB.load;
  17. import static com.rednum.graph.TiTanDB.query;
  18. import com.thinkaurelius.titan.core.EdgeLabel;
  19. import com.thinkaurelius.titan.core.Multiplicity;
  20. import com.thinkaurelius.titan.core.PropertyKey;
  21. import com.thinkaurelius.titan.core.TitanFactory;
  22. import com.thinkaurelius.titan.core.TitanGraph;
  23. import com.thinkaurelius.titan.core.TitanTransaction;
  24. import com.thinkaurelius.titan.core.attribute.Geoshape;
  25. import com.thinkaurelius.titan.core.attribute.Text;
  26. import com.thinkaurelius.titan.core.schema.ConsistencyModifier;
  27. import com.thinkaurelius.titan.core.schema.TitanGraphIndex;
  28. import com.thinkaurelius.titan.core.schema.TitanManagement;
  29. import com.thinkaurelius.titan.core.util.TitanCleanup;
  30. import java.io.BufferedReader;
  31. import java.io.FileReader;
  32. import java.io.IOException;
  33. import java.util.HashMap;
  34. import java.util.Iterator;
  35. import java.util.List;
  36. import java.util.Map;
  37. import java.util.Set;
  38. import org.apache.commons.configuration.BaseConfiguration;
  39. import org.apache.commons.configuration.Configuration;
  40. import org.apache.hadoop.hbase.Cell;
  41. import org.apache.hadoop.hbase.CellUtil;
  42. import org.apache.hadoop.hbase.HBaseConfiguration;
  43. import org.apache.hadoop.hbase.HConstants;
  44. import org.apache.hadoop.hbase.client.Get;
  45. import org.apache.hadoop.hbase.client.HTable;
  46. import org.apache.hadoop.hbase.client.Result;
  47. import org.apache.hadoop.hbase.client.ResultScanner;
  48. import org.apache.hadoop.hbase.client.Scan;
  49. import org.apache.hadoop.hbase.util.Bytes;
  50. import org.apache.tinkerpop.gremlin.process.traversal.Order;
  51. import org.apache.tinkerpop.gremlin.process.traversal.P;
  52. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
  53. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
  54. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
  55. import org.apache.tinkerpop.gremlin.structure.Direction;
  56. import org.apache.tinkerpop.gremlin.structure.Edge;
  57. import org.apache.tinkerpop.gremlin.structure.T;
  58. import org.apache.tinkerpop.gremlin.structure.Vertex;
  59. import org.codehaus.jettison.json.JSONArray;
  60. import org.codehaus.jettison.json.JSONObject;
  61. /**
  62. *
  63. * @author X.H.Yang
  64. */
  65. public class TiTanNew {
  66. public static final String INDEX_NAME = "search";
  67. public static org.apache.hadoop.conf.Configuration conf = null;
  68. static {
  69. conf = HBaseConfiguration.create();
  70. conf.setLong(HConstants.HBASE_REGIONSERVER_LEASE_PERIOD_KEY, 180000);
  71. }
  72. public static TitanGraph create() {
  73. try {
  74. //创建名为newgraph的表。
  75. TitanGraph graph = TitanFactory.build()
  76. .set("storage.backend", "hbase")
  77. .set("storage.hostname", "192.168.1.252")
  78. .set("storage.hbase.table", "newgraph")
  79. .set("cache.db-cache", "true")
  80. .set("cache.db-cache-clean-wait", "20")
  81. .set("cache.db-cache-time", "180000")
  82. .set("cache.db-cache-size", "0.5")
  83. .set("index.newgraph.backend", "elasticsearch")
  84. .set("index.newgraph.hostname", "192.168.1.212")
  85. .set("index.newgraph.port", 9300)
  86. .set("index.newgraph.elasticsearch.client-only", true)
  87. .open();
  88. return graph;
  89. } catch (Exception e) {
  90. System.out.println(e);
  91. return null;
  92. }
  93. }
  94. public static void loadWithoutMixedIndex(final TitanGraph graph,
  95. boolean uniqueNameCompositeIndex) {
  96. load(graph, null, uniqueNameCompositeIndex);
  97. }
  98. public static void load(final TitanGraph graph) {
  99. load(graph, "newgraph", true);
  100. }
  101. public static void load(final TitanGraph graph, String mixedIndexName,
  102. boolean uniqueNameCompositeIndex) {
  103. // 添加索引,name为key,并建立边和点的联合索引,提高查询速率
  104. try {
  105. TitanManagement mgmt = graph.openManagement();
  106. final PropertyKey name = mgmt.makePropertyKey("name").dataType(String.class).make();
  107. TitanManagement.IndexBuilder nameIndexBuilder = mgmt.buildIndex("name", Vertex.class).addKey(name);
  108. if (uniqueNameCompositeIndex) {
  109. nameIndexBuilder.unique();
  110. }
  111. TitanGraphIndex namei = nameIndexBuilder.buildCompositeIndex();
  112. mgmt.setConsistency(namei, ConsistencyModifier.LOCK);
  113. final PropertyKey group = mgmt.makePropertyKey("group").dataType(Integer.class).make();
  114. if (null != mixedIndexName) {
  115. mgmt.buildIndex("vertices", Vertex.class).addKey(group).buildMixedIndex(mixedIndexName);
  116. }
  117. final PropertyKey projectname = mgmt.makePropertyKey("projectname").dataType(String.class).make();
  118. final PropertyKey sectionname = mgmt.makePropertyKey("sectionname").dataType(String.class).make();
  119. if (null != mixedIndexName) {
  120. mgmt.buildIndex("edges", Edge.class).addKey(projectname).addKey(sectionname).buildMixedIndex(mixedIndexName);
  121. }
  122. mgmt.makeEdgeLabel("ComSec").multiplicity(Multiplicity.MANY2ONE).make();
  123. mgmt.makeEdgeLabel("SecPro").multiplicity(Multiplicity.MANY2ONE).make();
  124. mgmt.commit();
  125. } catch (Exception e) {
  126. System.out.println(e);
  127. }
  128. }
  129. public void doCreatCompanyGraph() {
  130. TitanGraph graph = create(); //建立表
  131. load(graph); //添加索引
  132. GraphTraversalSource g = graph.traversal(); //遍历表
  133. String error = "";
  134. while (true) {
  135. try {
  136. //遍历采集的数据CompanyInfos_test表中所有公司名字
  137. HTable country = new HTable(conf, "CompanyInfos_test");
  138. Scan s = new Scan();
  139. ResultScanner rs = country.getScanner(s);
  140. int count = 0;
  141. for (Result r : rs) {
  142. count++;
  143. String row = Bytes.toString(r.getRow());
  144. error = row;
  145. //避免同一公司重复加入图谱,name设为索引key,只能唯一存在
  146. if (g.V().has("group", 1).has("name", row).hasNext()) {
  147. continue;
  148. }
  149. //建立公司名称节点,属性有label, name, group, 即公司名字的group为1,label为company
  150. Vertex v1 = graph.addVertex(T.label, "company", "name", row, "group", 1);
  151. //Hbase相关查询语句,根据rowkey获取指定列族里所有列
  152. Get get = new Get(Bytes.toBytes(row));
  153. get.addFamily(Bytes.toBytes("performance_owner"));
  154. Result rs1 = country.get(get);
  155. for (Cell cell : rs1.rawCells()) {
  156. String performance = "";
  157. String project = "";
  158. performance = Bytes.toString(CellUtil.cloneValue(cell));
  159. project = getProjectByPer(performance);
  160. if (!"".equals(performance)) {
  161. performance = performance.replace(row, "");
  162. }
  163. if (!"".equals(project)) {
  164. performance = performance.replace(project, "");
  165. }
  166. if ("/".equals(performance) || "".equals(performance)) {
  167. performance = project;
  168. }
  169. Vertex v2 = null;
  170. if (!"".equals(performance)) {
  171. //建立标段节点,属性有label, sectionname, group, 即标段的group为2,label为section
  172. v2 = graph.addVertex(T.label, "section", "sectionname", performance, "group", 2);
  173. //添加一条由v1指向v2,属性为ComSec的边
  174. v1.addEdge("ComSec", v2);
  175. }
  176. if (!"".equals(project)) {
  177. Vertex v3 = null;
  178. if (g.V().has("group", 3).has("projectname", project).hasNext()) {
  179. v3 = g.V().has("group", 3).has("projectname", project).next();
  180. } else {
  181. v3 = graph.addVertex(T.label, "project", "projectname", project, "group", 3);
  182. }
  183. v2.addEdge("SecPro", v3);
  184. }
  185. graph.tx().commit();
  186. }
  187. graph.tx().commit();
  188. System.out.println(row + ": 第" + count + "家公司");
  189. }
  190. rs.close();
  191. System.out.println("共有数据" + count);
  192. break;
  193. } catch (Exception e) {
  194. System.out.println(e.toString());
  195. System.out.println("公司:" + error + "捕获异常");
  196. //对异常的结点进行删除
  197. Vertex ver = g.V().has("group", 1).has("name", error).next();
  198. GraphTraversal<Vertex, Vertex> mF = g.V(ver).out("ComSec");
  199. while (mF.hasNext()) {
  200. Vertex ver1 = mF.next();
  201. g.V(ver1).drop().iterate(); //删除该节点
  202. }
  203. g.V(ver).drop().iterate();
  204. graph.tx().commit();
  205. continue;
  206. }
  207. }
  208. System.out.println(g.V().count().next());
  209. System.out.println(g.E().count().next());
  210. graph.close();
  211. }
  212. public String getProjectByPer(String per) throws IOException {
  213. HTable table = new HTable(conf, "CompanyOwner_test");
  214. Get get = new Get(per.getBytes("utf-8"));
  215. get.addColumn(Bytes.toBytes("ProjectInfo"), Bytes.toBytes("ProjectName"));
  216. Result rs = table.get(get);
  217. String project = "";
  218. for (Cell cell : rs.rawCells()) {
  219. project = Bytes.toString(CellUtil.cloneValue(cell));
  220. }
  221. return project;
  222. }
  223. public static void main(String[] args) throws Exception {
  224. try {
  225. TiTanNew titan = new TiTanNew();
  226. titan.doCreatCompanyGraph();
  227. } catch (Exception e) {
  228. e.toString();
  229. }
  230. }
  231. }

2、关系检索

下面给出模糊搜索项目:“遵义至毕节高速公路”的实现过程。

 
 
  1. /*
  2. * To change this license header, choose License Headers in Project Properties.
  3. * To change this template file, choose Tools | Templates
  4. * and open the template in the editor.
  5. */
  6. package com.rednum.graph;
  7. import com.google.gson.Gson;
  8. import com.thinkaurelius.titan.core.TitanException;
  9. import com.thinkaurelius.titan.core.TitanFactory;
  10. import com.thinkaurelius.titan.core.TitanGraph;
  11. import com.thinkaurelius.titan.core.TitanVertex;
  12. import com.thinkaurelius.titan.core.attribute.Text;
  13. import java.util.ArrayList;
  14. import java.util.HashMap;
  15. import java.util.LinkedHashMap;
  16. import java.util.List;
  17. import java.util.Map;
  18. import java.util.NoSuchElementException;
  19. import java.util.Set;
  20. import org.apache.commons.configuration.BaseConfiguration;
  21. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
  22. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
  23. import org.apache.tinkerpop.gremlin.structure.Direction;
  24. import org.apache.tinkerpop.gremlin.structure.Vertex;
  25. /**
  26. *
  27. * @author X.H.Yang
  28. */
  29. public class CompanyCountrySearch {
  30. private Gson gson = new Gson();
  31. public static TitanGraph open() {
  32. try {
  33. org.apache.commons.configuration.Configuration conf = new BaseConfiguration();
  34. conf.setProperty("storage.backend", "hbase");
  35. conf.setProperty("storage.hostname", "192.168.1.252");
  36. conf.setProperty("storage.hbase.table", "newgraph");
  37. TitanGraph graph = TitanFactory.open(conf);
  38. return graph;
  39. } catch (Exception e) {
  40. System.out.println(e);
  41. return null;
  42. }
  43. }
  44. public String doSearch() {
  45. TitanGraph graph = CompanyCountrySearch.open();
  46. GraphTraversalSource g = graph.traversal();
  47. try {
  48. String jstr = "";
  49. Iterable<TitanVertex> mm = graph.query().has("group", 3).has("projectname", Text.REGEX, ".*遵义至毕节高速公路.*").vertices();
  50. for (TitanVertex tt : mm) {
  51. HashMap<String, Object> params = new HashMap<>();
  52. List<Map> nodes = new ArrayList<>();
  53. List<Object> links = new ArrayList<>();
  54. Map map = new LinkedHashMap();
  55. GraphTraversal<Vertex, Vertex> mF = g.V(tt).in("SecPro");
  56. String project = g.V(tt).next().value("projectname");
  57. System.out.println("工程名称:" + project);
  58. map.put("name", project);
  59. map.put("group", 1);
  60. map.put("index", nodes.size());
  61. nodes.add(map);
  62. int i = 0;
  63. while (mF.hasNext()) {
  64. Vertex v1 = mF.next();
  65. String section = g.V(v1).next().value("sectionname");
  66. Vertex v2 = g.V(v1).in("ComSec").next();
  67. String company = g.V(v2).next().value("name");
  68. Map map1 = new LinkedHashMap();
  69. map1.put("name", section);
  70. map1.put("group", 2);
  71. map1.put("index", nodes.size());
  72. nodes.add(map1);
  73. i = i+1;
  74. Map link1 = new LinkedHashMap();
  75. link1.put("source", nodes.size()-1);
  76. link1.put("target", 0);
  77. links.add(link1);
  78. int temp = 0;
  79. int kk = 0;
  80. for (Map m : nodes) {
  81. if (m.containsValue(company)) {
  82. kk = (int) m.get("index");
  83. temp = 1;
  84. break;
  85. }
  86. }
  87. if (temp == 1) {
  88. Map templink = new LinkedHashMap();
  89. templink.put("source", kk);
  90. templink.put("target", nodes.size()-1);
  91. links.add(templink);
  92. } else {
  93. Map map2 = new LinkedHashMap();
  94. map2.put("name", company);
  95. map2.put("group", 3);
  96. map2.put("index", nodes.size());
  97. nodes.add(map2);
  98. Map link2 = new LinkedHashMap();
  99. link2.put("source", nodes.size()-1 );
  100. link2.put("target", nodes.size()-2);
  101. links.add(link2);
  102. i = i + 1;
  103. System.out.println("----------");
  104. GraphTraversal<Vertex, Vertex> gt = g.V(v2).out("ComSec");
  105. int j = 0;
  106. int count = nodes.size()-1;
  107. while (gt.hasNext()) {
  108. Vertex v3 = gt.next();
  109. String section1 = g.V(v3).next().value("sectionname");
  110. String project1 = g.V(v3).out("SecPro").next().value("projectname");
  111. if (project1.equals(project)) {
  112. continue;
  113. }
  114. Map map11 = new LinkedHashMap();
  115. map11.put("name", section1);
  116. map11.put("group", 2);
  117. map11.put("index", nodes.size());
  118. nodes.add(map11);
  119. j = j +1;
  120. Map link11 = new LinkedHashMap();
  121. link11.put("source", count);
  122. link11.put("target", nodes.size()-1);
  123. links.add(link11);
  124. int temp1 = 0;
  125. int kk1 = 0;
  126. for (Map m : nodes) {
  127. if (m.containsValue(project1)) {
  128. kk1 = (int) m.get("index");
  129. temp1 = 1;
  130. break;
  131. }
  132. }
  133. if (temp1 == 1) {
  134. Map link22 = new LinkedHashMap();
  135. link22.put("source", nodes.size()-1);
  136. link22.put("target", kk1);
  137. links.add(link22);
  138. } else {
  139. Map map22 = new LinkedHashMap();
  140. map22.put("name", project1);
  141. map22.put("group", 1);
  142. map22.put("index", nodes.size());
  143. nodes.add(map22);
  144. Map link22 = new LinkedHashMap();
  145. link22.put("source", nodes.size()-2);
  146. link22.put("target", nodes.size()-1);
  147. links.add(link22);
  148. j = j + 1;
  149. }
  150. }
  151. i = i +j + 2;
  152. }
  153. }
  154. params.put("nodes", nodes);
  155. params.put("links", links);
  156. jstr = gson.toJson(params);
  157. System.out.println(jstr);
  158. break;
  159. }
  160. graph.close();
  161. return jstr;
  162. } catch (NoSuchElementException | TitanException e) {
  163. e.toString();
  164. System.out.println(e.toString());
  165. return null;
  166. }
  167. }
  168. public static void main(String[] args) {
  169. CompanyCountrySearch search = new CompanyCountrySearch();
  170. search.doSearch();
  171. }
  172. }

我们也可根据公司名字检索出相关联的项目和标段信息,代码如下:

 
 
  1. /*
  2. * To change this license header, choose License Headers in Project Properties.
  3. * To change this template file, choose Tools | Templates
  4. * and open the template in the editor.
  5. */
  6. package com.rednum.graph;
  7. import com.google.gson.Gson;
  8. import com.thinkaurelius.titan.core.TitanFactory;
  9. import com.thinkaurelius.titan.core.TitanGraph;
  10. import java.util.ArrayList;
  11. import java.util.HashMap;
  12. import java.util.LinkedHashMap;
  13. import java.util.List;
  14. import java.util.Map;
  15. import org.apache.commons.configuration.BaseConfiguration;
  16. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
  17. import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
  18. import org.apache.tinkerpop.gremlin.structure.Vertex;
  19. /**
  20. *
  21. * @author X.H.Yang
  22. */
  23. public class TitanSearchByCompany {
  24. private Gson gson = new Gson();
  25. public static TitanGraph open() {
  26. try {
  27. org.apache.commons.configuration.Configuration conf = new BaseConfiguration();
  28. conf.setProperty("storage.backend", "hbase");
  29. conf.setProperty("storage.hostname", "192.168.1.252");
  30. conf.setProperty("storage.hbase.table", "newgraph");
  31. TitanGraph graph = TitanFactory.open(conf);
  32. return graph;
  33. } catch (Exception e) {
  34. System.out.println(e);
  35. return null;
  36. }
  37. }
  38. public String searchBycom() {
  39. TitanGraph graph = open();
  40. GraphTraversalSource g = graph.traversal();
  41. try {
  42. HashMap<String, Object> params = new HashMap<>();
  43. List<Map> nodes = new ArrayList<>();
  44. List<Object> links = new ArrayList<>();
  45. GraphTraversal<Vertex, Vertex> mF = g.V().has("group", 1).has("name", "中铁十局集团有限公司").out("ComSec");
  46. Map map = new LinkedHashMap();
  47. map.put("name", "中铁十局集团有限公司");
  48. map.put("group", 1);
  49. map.put("index", nodes.size());
  50. nodes.add(map);
  51. int count = 0;
  52. while (mF.hasNext()) {
  53. Vertex v1 = mF.next();
  54. String section = g.V(v1).next().value("sectionname");
  55. Map map1 = new LinkedHashMap();
  56. map1.put("name", section);
  57. map1.put("group", 2);
  58. map1.put("index", nodes.size());
  59. nodes.add(map1);
  60. Map link1 = new LinkedHashMap();
  61. link1.put("source", 0);
  62. link1.put("target", nodes.size() - 1);
  63. links.add(link1);
  64. count = nodes.size() - 1;
  65. Vertex v2 = g.V(v1).out("SecPro").next();
  66. String project = g.V(v2).next().value("projectname");
  67. int temp = 0;
  68. int kk = 0;
  69. for (Map m : nodes) {
  70. if (m.containsValue(project)) {
  71. kk = (int) m.get("index");
  72. temp = 1;
  73. break;
  74. }
  75. }
  76. if (temp == 1) {
  77. Map templink = new LinkedHashMap();
  78. templink.put("source", nodes.size() - 1);
  79. templink.put("target", kk);
  80. links.add(templink);
  81. } else {
  82. Map map2 = new LinkedHashMap();
  83. map2.put("name", project);
  84. map2.put("group", 3);
  85. map2.put("index", nodes.size());
  86. nodes.add(map2);
  87. Map link2 = new LinkedHashMap();
  88. link2.put("source", count);
  89. link2.put("target", nodes.size() - 1);
  90. links.add(link2);
  91. }
  92. }
  93. params.put("nodes", nodes);
  94. params.put("links", links);
  95. String jstr = gson.toJson(params);
  96. System.out.println(jstr);
  97. graph.close();
  98. return jstr;
  99. } catch (Exception e) {
  100. e.toString();
  101. return null;
  102. }
  103. }
  104. public static void main(String[] args) {
  105. TitanSearchByCompany search = new TitanSearchByCompany();
  106. String result = search.searchBycom();
  107. }
  108. }

成果展示

由于Titan数据库没有可视化界面,所以我们在web平台上开发了根据搜索内容,呈现关系图谱的功能,图的展示主要用了D3中的力导向图,力导向图的实现将在下一篇文中由前端同事介绍。

通过下面的搜索界面,得到公司或者项目的业绩图谱。
Titan数据库快速入门之决战云巅
搜索“中铁十局集团有限公司”得到的关系图为:(效果图是不是很炫)
Titan数据库快速入门之决战云巅

搜索项目“雅安至康定”得到下面的关系图谱,由于屏幕有限,只展示一层的关系。
Titan数据库快速入门之决战云巅

文章到此结束,这是小编学了Titan数据库后完成的第一个项目,如有不对的地方,请在下方留言指正。感兴趣的我们可以相互交流。

本文来源水边风,由架构君转载发布,观点不代表Java架构师必看的立场,转载请标明来源出处:https://javajgs.com/archives/16305

发表评论